Really odd one: parts of global DNS just dropped off the map

[Andy Holyer]

I've never seen this one ever before, and I don't even really know
where to ask.

Apologies if this is off-topic, but here goes.

I administer the primary DNS for a bunch of domains which our company
hosts, plus mail and DNS for the customers of our ISP. The primary DNS
server is a Dell Poweredge running FreeBSD hosted at Telehouse in
London's docklands.

Some point yesterday afternoon (GMT). I started getting reports that
certain sites were failing on DNS. Notably www.nasa.gov, which my boss
uses to demonstrate that broadband is operating. The fact that he
discovered this five minutes before I was planning to go home, and
when he couldn't leave a client's site because they were not
distinguishing between bad DNS and lack of connectivity helped none of
our tempers in the matter.

It seemed a bunch of (mainly US-based) sites were failing DNS. Other
(UK-based) worked fine. Trying another server in the same facility
gave the same result. Finally I switced forwarders to another ISP and
called it a day.

This morning I switched things back and all appears fine. however
digging around, I don't get ping response from about half the hosts in
named.root. b.root-servers.net, for example. Now, I can understand
that root servers would just turn off ICMP echo since they're busy
enough as it is, but it still worries me a bit.

I've never seen this sort of behaviour before, and I'm not at all sure
where to start in finding out what's going on, and whether there's
some subtle mis-configuration on my part. From my part, the serial
number in my root db files tells me that I haven't touched the named
config since early June, so I would be most surprised if it was
functioning fine, barely idling, and would then just go haywire like
that.

*Any* advice as to where I could go from here to ensure integrity of
DNS is most gratefully recieved.

---

Andy Holyer, Hedgehog Broadband, Brighton UK.