Queries to a positively cached zone are failing (phila.gov)

[Greg Chavez]

I work at a large .gov gateway whose BIND servers cannot resolve any
queries for phila.gov.  I see bad domains all the time, and a quick
dump of the cache and a dig here and a dig there usually point to one
or more bad name servers.  I've had this problem in the past with
phila.gov, a zone that our mail servers hit very often; usually, our
mail queues will get a little high until our forwarders replace their
caches with a fresh iterative query to the zone's working name server.
 Life goes on.

We are experiencing a total phila.gov blackout right now.  All queries
for it time out.  But this time, we have both of phila.gov's name
servers in our cache with glue:

# grep -i phila.gov named_dump.db
phila.GOV.              85957   NS      DNS.phila.gov.
                        85957   NS      DNS2.phila.gov.
DNS.phila.GOV.          85957   A       170.115.249.10
DNS2.phila.GOV.         85957   A       170.115.249.11

If I do digs @ either NS IP, I get answers. Digs using my forwarders
time out.  Dig traces get me the NS records for the dot-gov servers

Clearing my cache has no effect.  I am utterly stumped... everything I
have ever seen before tells me that my name server *should* be seeing
this domain.  What awful assumption(s) are keeping me from seeing the
problem?

Bind is a bit crusty: 9.2.2p3.

--
--Greg Chavez
--