Failing DNS Server Diagnostic Help Requested

[Tim Daneliuk]

Environment:  Master/Slave with Split Horizon both on FreeBSD-STABLE
              Bind 9.16.24_1
              Master out in a cloud server
              Slave on a physical server with a static IP on Comcast Business

Problem:  After years of stable behavior, Slave intermittently not resolving
          addresses a few months ago, and then completely stopped working
          yesterday. We also noticed that the Slave will not update its files
          upon notify from the Master.

Action Taken: Replaced Slave with a clone of the Master instance.  That new
              Master does properly resolve names inside our zone, whether
              the requestor is on our LAN our one of our trusted servers out
              on the internet that are allowed to see internal names.

              HOWEVER, that new master instance will not resolve names in
              zones other than ours.  We're working around this by
              forwarding these failed lookups to our original master -
              that is working fine.

              So, we have two masters with the same configuration and
              tables, but one resolves outside names and one does not.
              We've tried disabling DNSSEC validation and opening up our
              firewalls and got nowhere.

              When the lookups outside our zone fail, we see this:

13-Jan-2022 14:28:09.702 resolver: notice: DNS format error from 192.203.230.10#53 resolving ./NS for <unknown>: non-improving referral
13-Jan-2022 14:28:09.702 lame-servers: info: FORMERR resolving './NS/IN': 192.203.230.10#53
13-Jan-2022 14:28:09.721 resolver: notice: DNS format error from 192.36.148.17#53 resolving ./NS for <unknown>: non-improving referral
13-Jan-2022 14:28:09.721 lame-servers: info: FORMERR resolving './NS/IN': 192.36.148.17#53
13-Jan-2022 14:28:09.741 resolver: notice: DNS format error from 193.0.14.129#53 resolving ./NS for <unknown>: non-improving referral
13-Jan-2022 14:28:09.741 lame-servers: info: FORMERR resolving './NS/IN': 193.0.14.129#53
13-Jan-2022 14:28:09.763 resolver: notice: DNS format error from 199.7.91.13#53 resolving ./NS for <unknown>: non-improving referral
13-Jan-2022 14:28:09.763 lame-servers: info: FORMERR resolving './NS/IN': 199.7.91.13#53
13-Jan-2022 14:28:09.781 resolver: notice: DNS format error from 202.12.27.33#53 resolving ./NS for <unknown>: non-improving referral
13-Jan-2022 14:28:09.781 lame-servers: info: FORMERR resolving './NS/IN': 202.12.27.33#53
13-Jan-2022 14:28:09.801 resolver: notice: DNS format error from 199.7.83.42#53 resolving ./NS for <unknown>: non-improving referral
13-Jan-2022 14:28:09.801 lame-servers: info: FORMERR resolving './NS/IN': 199.7.83.42#53
13-Jan-2022 14:28:09.820 resolver: notice: DNS format error from 192.58.128.30#53 resolving ./NS for <unknown>: non-improving referral
13-Jan-2022 14:28:09.820 lame-servers: info: FORMERR resolving './NS/IN': 192.58.128.30#53
13-Jan-2022 14:28:09.837 resolver: notice: DNS format error from 198.41.0.4#53 resolving ./NS for <unknown>: non-improving referral
13-Jan-2022 14:28:09.837 lame-servers: info: FORMERR resolving './NS/IN': 198.41.0.4#53
13-Jan-2022 14:28:09.855 resolver: notice: DNS format error from 198.97.190.53#53 resolving ./NS for <unknown>: non-improving referral
13-Jan-2022 14:28:09.855 lame-servers: info: FORMERR resolving './NS/IN': 198.97.190.53#53
13-Jan-2022 14:28:09.875 resolver: notice: DNS format error from 192.5.5.241#53 resolving ./NS for <unknown>: non-improving referral
13-Jan-2022 14:28:09.875 lame-servers: info: FORMERR resolving './NS/IN': 192.5.5.241#53
13-Jan-2022 14:28:09.893 resolver: notice: DNS format error from 192.112.36.4#53 resolving ./NS for <unknown>: non-improving referral
13-Jan-2022 14:28:09.893 lame-servers: info: FORMERR resolving './NS/IN': 192.112.36.4#53
13-Jan-2022 14:28:09.921 resolver: notice: DNS format error from 199.9.14.201#53 resolving ./NS for <unknown>: non-improving referral
13-Jan-2022 14:28:09.921 lame-servers: info: FORMERR resolving './NS/IN': 199.9.14.201#53
13-Jan-2022 14:28:09.937 resolver: notice: DNS format error from 192.33.4.12#53 resolving ./NS for <unknown>: non-improving referral
13-Jan-2022 14:28:09.937 lame-servers: info: FORMERR resolving './NS/IN': 192.33.4.12#53
13-Jan-2022 14:28:09.938 resolver: info: resolver priming query complete


So ... could this be Comcast munging about in the DNS traffic?   Other suggestions
of where to look appreciated as well ...


(We have a fair bit of other logging data to be examined, I just didn't want to
spam the whole list with all that ...)


-- 
----------------------------------------------------------------------------
Tim Daneliuk     tundra at tundraware.com
PGP Key:         http://www.tundraware.com/PGP/