reverse lookup behavior change

[coolburn95 at yahoo.com]

Hello,

We have an internal network of the form 10.x.x.x

We have two DNS servers that are multi-homed, but are used by the
internal network at 10.0.0.10 and 10.0.0.11

All of the internal servers have resolv.conf setup as:
nameserver 10.0.0.10
nameserver 10.0.0.11

This has worked perfectly for 8 months.

Today, we suddenly started getting timeouts in our application server
connecting to our db server.  Then, I saw I was also getting very slow
times to connect using SSH.  I knew right away this was DNS related.

The DNS servers were still responding perfectly to requests ("dig
google.com" worked fine on all internal servers, and was getting
responses from both 10.0.0.10 and 10.0.0.11)

The problem is, they were giving extremely slow responses to "dig -x
10.0.0.5", etc.

I realize now that the problem is, we had no reverse DNS defined for
these internal IP addreses.  I simply added them into the hosts file on
the servers that needed them, and connection times between our
applications, and ssh connection times became instant.

My question is, why did we only just now start noticing this behavior?
To be honest, I hadn't tried a "dig -x 10.0.0.5" in a long, long time
on any of the internal servers.  But, I ssh among them throughout the
day, every day.  Our applications had no problems connecting, etc.

My only thought is that our two DNS servers somehow had the failed
reverse lookup cached, and today decided to do live lookups on them.
What I don't get, is why they didn't seem to be cacheing the failed
lookups anymore.

Repeated lookups of "dig -x 10.0.0.5" result in:
[root at db01 etc]# dig -x 10.0.0.5

; <<>> DiG 9.2.4 <<>> -x 10.0.0.5
;; global options:  printcmd
;; connection timed out; no servers could be reached
[root at db01 etc]#

Whereas:
[root at db01 etc]# dig -x 216.239.39.99

; <<>> DiG 9.2.4 <<>> -x 216.239.39.99
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 46492
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;99.39.239.216.in-addr.arpa.    IN      PTR

;; AUTHORITY SECTION:
39.239.216.in-addr.arpa. 54     IN      SOA     ns1.google.com.
dns-admin.google.com. 2004031201 21600 3600 1038800 60

;; Query time: 1 msec
;; SERVER: 10.0.0.10#53(10.0.0.10)
;; WHEN: Thu May 12 13:03:03 2005
;; MSG SIZE  rcvd: 104

Any insight into this is much appreciated!

Thank you,
-Michael Thomas