Hell breaks loose in the afternoon with format error from X.X.X.X#53 resolving ./NS: non-improving referral

[Mark Andrews]

It’s a long known issue with so called “Transparent” DNS proxies/accelerators/firewalls.  Iterative resolvers expect to talk to authoritative servers.  They ask questions differently to the way they do when they talk to a recursive server.  Answers from different levels of the DNS hierarchy for the same question are different.  If you just cache and return the previous answer you break iterative lookups.  The answers from recursive servers are different to those from authoritative servers.

You get the same sort of problem in many hotels if you have an iterative resolver on your portable devices.  Switching named to use a public recursive server that supports DNSSEC in forward only mode helps sometimes.  It really depends on what the middleware is doing.

Mark

> On 6 May 2022, at 09:35, Ted Mittelstaedt <tedm at ipinc.net> wrote:
> 
> Thought I would document this in case anyone else gets bit by it
> 
> I have several nameservers and other servers on a Comcast copper connection  (cable internet) in the office using a Technicolor Business Router CGA4131COM  modem.  This is Comcast's de-facto standard modem as of 2022 for business connections in the western half of the US (maybe countrywide)
> 
> I have a ticket with Comcast open on another issue that was escalated to
> second tier.  Well some bozo in second tier finally gets around to working it and decides to login to the Technicolor and sees that the
> firewall is turned off, and so decides to helpfully "fix" the problem
> by turning it back on.
> 
> So there I am driving along, miles away, minding my own business then all the sudden unknown to me in the office all DNS lookups fail, mailservers on the circuit start spewing, and at the same time my cell phone rings with some tech from Comcast brightly chirping how she "fixed" the problem.
> 
> Of course as icing on the cake when I pull over to deal with it I'm in an area with so poor cell signal I can't even get an internet connection up from my laptop.
> 
> By the time I get back to the office, discover what was going on, call back into them, and have them reverse what was done the rest of the afternoon was scotched and I was pissed!
> 
> Nearest sort-of explanation I could find was much handwaving and speculation in the following:
> 
> https://serverfault.com/questions/489010/bind-formerr-errors-in-syslog
> 
> Anyway, it seems clear that the Technicolor's firewall, when enabled,
> transparently DOES intercept DNS queries to answer them out of a cache
> on the router, which has the effect of completely scotching the ability
> of a nameserver to do recursive queries.  My syslog logs were filling up
> and rolling over in less than 2 hours with thousands of these referral
> errors.
> 
> The serverfault seems to think that this kind of thing is due to possible bugs in bind but the moment the modem was reconfigured to turn
> off the firewall the log entries stopped.
> 
> I'm not keen on further experimentation on this, I just wanted to post it in case someone else is dealing with inexplicable errors and pulling their hair out.
> 
> Ted
> -- 
> Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list
> 
> ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.
> 
> 
> bind-users mailing list
> bind-users at lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742              INTERNET: marka at isc.org