Hell breaks loose in the afternoon with format error from X.X.X.X#53 resolving ./NS: non-improving referral

[Ted Mittelstaedt]

Thought I would document this in case anyone else gets bit by it

I have several nameservers and other servers on a Comcast copper 
connection  (cable internet) in the office using a Technicolor Business 
Router CGA4131COM  modem.  This is Comcast's de-facto standard modem as 
of 2022 for business connections in the western half of the US (maybe 
countrywide)

I have a ticket with Comcast open on another issue that was escalated to
second tier.  Well some bozo in second tier finally gets around to 
working it and decides to login to the Technicolor and sees that the
firewall is turned off, and so decides to helpfully "fix" the problem
by turning it back on.

So there I am driving along, miles away, minding my own business then 
all the sudden unknown to me in the office all DNS lookups fail, 
mailservers on the circuit start spewing, and at the same time my cell 
phone rings with some tech from Comcast brightly chirping how she 
"fixed" the problem.

Of course as icing on the cake when I pull over to deal with it I'm in 
an area with so poor cell signal I can't even get an internet connection 
up from my laptop.

By the time I get back to the office, discover what was going on, call 
back into them, and have them reverse what was done the rest of the 
afternoon was scotched and I was pissed!

Nearest sort-of explanation I could find was much handwaving and 
speculation in the following:

https://serverfault.com/questions/489010/bind-formerr-errors-in-syslog

Anyway, it seems clear that the Technicolor's firewall, when enabled,
transparently DOES intercept DNS queries to answer them out of a cache
on the router, which has the effect of completely scotching the ability
of a nameserver to do recursive queries.  My syslog logs were filling up
and rolling over in less than 2 hours with thousands of these referral
errors.

The serverfault seems to think that this kind of thing is due to 
possible bugs in bind but the moment the modem was reconfigured to turn
off the firewall the log entries stopped.

I'm not keen on further experimentation on this, I just wanted to post 
it in case someone else is dealing with inexplicable errors and pulling 
their hair out.

Ted