Noisy messages from BIND about root hints change

[Cathy Almond]

On 07/01/13 17:14, Chris Thompson wrote:
> One (but only one) of our recursive nameservers, running BIND 9.8.3-P4
> we got a whole lot of messages in the log as a result of last week's change
> of address for d.root-servers.net:
> 
> Jan  4 06:24:08 recdns1.csx.cam.ac.uk named[9496]: general: warning:
>   checkhints: d.root-servers.net/A (128.8.10.90) missing from hints
> Jan  4 06:24:08 recdns1.csx.cam.ac.uk named[9496]: general: warning:
>   checkhints: d.root-servers.net/A (199.7.91.13) extra record in hints
> Jan  4 06:24:09 recdns1.csx.cam.ac.uk named[9496]: general: warning:
>   checkhints: d.root-servers.net/A (128.8.10.90) missing from hints
> Jan  4 06:24:09 recdns1.csx.cam.ac.uk named[9496]: general: warning:
>   checkhints: d.root-servers.net/A (199.7.91.13) extra record in hints
> 
> [... 1972 pairs of messages omitted ...]
> 
> Jan  4 08:50:05 recdns1.csx.cam.ac.uk named[9496]: general: warning:
>   checkhints: d.root-servers.net/A (128.8.10.90) missing from hints
> Jan  4 08:50:05 recdns1.csx.cam.ac.uk named[9496]: general: warning:
>   checkhints: d.root-servers.net/A (199.7.91.13) extra record in hints
> Jan  4 08:50:08 recdns1.csx.cam.ac.uk named[9496]: general: warning:
>   checkhints: d.root-servers.net/A (128.8.10.90) missing from hints
> Jan  4 08:50:08 recdns1.csx.cam.ac.uk named[9496]: general: warning:
>   checkhints: d.root-servers.net/A (199.7.91.13) extra record in hints
> 
> And then they stopped.
> 
> Now I can more or less work out what provoked the first message. We had
> already changed our root hints file the previous day (and done an rndc
> reconfig) but the old A record for d.root-servers.net was still in the
> cache (and was still there much later on 4 January as I explicitly did
> an rndc flushname on it for other reasons). One of our regular checking
> jobs at 06:24 will have used this recursive nameserver to look up the
> NS records for "." and the address records for the *.root-servers.net
> names so referenced.
> 
> But why did it keep going on and on about it? And what made it stop?
> Has anyone else seen anything similar?

I've seen one other report of repeating messages from checkhints - but
they also 'went away' (temporarily seen due to the transition of
addresses, and fixed by fixing the hints file to have D-root's new IPv4
address).

Differences between what's in the hints file and what's returned when
querying the root nameservers should only be 'spotted' by checkhints
when the cache is re-primed with the list of root nameservers - and that
should only happen when the roots have all expired from the cache.

What happens then is that the next time that a root nameserver needs to
be sent a query, named goes back to the hints and uses those to query
for an up-to-date list of root nameservers and their addresses - and
it's then that it will warn on any differences.

Now - on a busy cache, it would not be that unusual not to send queries
to root nameservers very often once you've been up and running for
awhile and have handled queries for all of the main TLDs.

So the theory I have for this is that the caches reporting a spate of
repeated warnings are ones in which there is a fairly conservative
max-cache-size set and then sufficient cache 'thrash' that the root
RRset is getting expired out of cache on the basis of 'least recently
used' (LRU cache management) to make space for other new entries.

Might that ring true in your case?

(Although - by 4th January - the new address should have been being
served by all the official root nameservers.  So it's still a bit odd
why you saw this at all, and moreover that you didn't see it before the
switch - so I'm not entirely convinced by the theory I'm putting forward
to you, and wonder if there was some other factor in play too).

Cathy