Stops doing reverses for a few hours

Tue Oct 7 18:59:23 UTC 2003

What version of bind or resolver are you using?
The below note from ARIN probably explains your problem, due to a change in 
their NS list last week.  I've seen some 8.2.x being affected.

-------- Original Message --------
Subject: Re: [ARIN-20031003.584] response to reverse PTR queries
Date: Mon, 06 Oct 2003 17:14:29 -0400 (EDT)
From: Cathy Murphy <cathym at arin.net>
To: Trent Arsenault

We've isolated the problem you've been experiencing to how the earlier BIND 
resolvers work. It appears that they do not properly follow referrals. The 
upshot is to get newer versions of your resolver software.
Here is what is happening:

   dig -x 129.159.39.15

  * Assuming no cache, a query goes to the root servers,
    which return no answer but give the name servers for
    129.in-addr.arpa (referral) to [chia,dill,...indigo].arin.net

  * A query then goes to the .net name servers
    [a,b...m].gtld-servers.net for [chia,dill...indigo].arin.net

This is where the problem arises. There is no A record (glue) for 
[chia...indigo].arin.net on the .net nameservers. Previously, our zones 
were also served by arrowroot.arin.net and buchu.arin.net, which had 
(inappropriate) glue records in .net. (You can still see them by doing "dig 
@a.gtld-servers.net arrowroot.arin.net".) We are phasing out these servers. 
On Thursday, we removed arrowroot and buchu from the delegations for our 
in-addr.arpa zones.  Although the change was done correctly it uncovered a 
hidden misconfiguration which enabled older BIND resolvers to work.

The return of arrowroot's A record in this way violated DNS rules on 
authority and credibility of data, facets better enforced in newer versions 
of BIND. Apparently older versions of BIND resolvers rely on this A record 
to descend the tree towards the answer and are unable to handle the 
referral that should have been returned.

So now, instead of responding with an A resource record, the .net name 
servers are correctly responding with a referral to the arin.net name 
servers: [a3,b3...].nstld.com. What the resolver should do is follow this 
referral to get the A record of the [chia,dill,etc].arin.net name servers. 
Instead, it appears to time out.

We haven't figured out what versions of BIND do and do not work with our 
changes.  We also haven't dug deeply enough to know why the older resolvers 
fail to pursue the referral messages.

I apologize if this seems like it's another case of "we're making a change 
and you have to live with it."  All of the options we've identified for 
avoiding this were kludges and hacks.  We couldn't come up with a way to 
help older resolvers without otherwise risking some other misconfiguration.

Regards,

Cathy Murphy
ARIN

 > Hello Cathy,
 >
 > We are still experiencing the reverse DNS problem and can reproduce the
 > problem where no data is returned from arin servers.
 >
 > I'm not sure what the problem is yet, but I can be available to 
reproduce the
 > problem from our side and someone can watch from the other end.  I'm
 > reachable today at my office.
 >
 > Best Regards,
 >
 > Trent Arsenault
 > trent.arsenault at sun.com
 >
 > Cathy Murphy wrote:
 >
 > > Hello,
 > >
 > > I just wanted to let you know that this matter has been forwarded on to me
 > > to investigate. I cannot replicate the problem, but am working to see what
 > > might have happened. Are you still experiencing these issues? If not, do
 > > you have an approximate time when it was resolved?
 > >
 > > Thank you.
 > >
 > > Cathy Murphy
 > > ARIN
 > > +1 703 227 9875 (Office)
 > > +1 571 436 6068 (Mobile)

Trent Arsenault
trent at trent.us

At 08:11 AM 10/7/2003, Tuc wrote:
> > >     That would make sense, but the first authoritative server is
> > >216.231.111.14, in the same /19 and the second is 209.51.161.86 which is
> > >offsite. We have checked all packet filers and allow-query and it should
> > >not be a problem. PLUS, the 2nd major time it happened, it stopped almost
> > >exactly 4 hours after it started with NO changes anywhere in our
> > >configurations or network.
> >
> > Have you tried using a sniffer to see the traffic between the servers when
> > this happens?
> >
>
>This is from a tcpdump atr the time it was happening:
>
>14:17:54.112038 216.231.119.95.53 > 198.41.0.4.53: 2760 PTR? 
>11.113.231.216.in-addr.arpa. (45) (ttl 64, id 5410)
>14:17:54.120434 198.41.0.4.53 > 216.231.119.95.53: 2760- q: 
>11.113.231.216.in-addr.arpa. 0/7/0 (198) (DF) (ttl 52, id 0)
>14:17:54.121001 192.55.83.30.53 > 216.231.119.95.53: 25408- q: 
>chia.ARIN.NET. 0/7/7 (271) (DF) (ttl 52, id 0)
>14:17:54.121303 192.55.83.30.53 > 216.231.119.95.53: 43090- q: 
>indigo.ARIN.NET. 0/7/7 (273) (DF) (ttl 52, id 0)
>14:17:54.121335 192.55.83.30.53 > 216.231.119.95.53: 65208- q: 
>ginseng.ARIN.NET. 0/7/7 (274) (DF) (ttl 52, id 0)
>14:17:54.122500 192.55.83.30.53 > 216.231.119.95.53: 6737- q: 
>figwort.ARIN.NET. 0/7/7 (274) (DF) (ttl 52, id 0)
>14:17:54.122792 192.55.83.30.53 > 216.231.119.95.53: 152- q: 
>henna.ARIN.NET. 0/7/7 (272) (DF) (ttl 52, id 0)
>14:17:54.125389 192.55.83.30.53 > 216.231.119.95.53: 47761- q: 
>dill.ARIN.NET. 0/7/7 (271) (DF) (ttl 52, id 0)
>14:17:54.125624 192.55.83.30.53 > 216.231.119.95.53: 12871- q: 
>epazote.ARIN.NET. 0/7/7 (274) (DF) (ttl 52, id 0)
>
>Which is where it appears to hang on that request, nothing further in the
>tcpdump related to that query...when it started working later on, it was
>querying the secondary server for the in-addr and not the primary, not sure
>if that's related or just coincidence:
>
>14:43:08.731324 216.231.119.95.53 > 209.51.161.86.53: 398 PTR? 
>11.113.231.216.in-addr.arpa. (45) (ttl 64, id 52493)
>14:43:08.732410 209.51.161.86.53 > 216.231.119.95.53: 398* q: 
>11.113.231.216.in-addr.arpa. 1/2/2 11.113.231.216.in-addr.arpa. PTR ti
>gger.tolim.net. (158) (ttl 60, id 43289)
>
>It appears to be working now in general, not sure if ARIN changed things back
>on their end or what the triggering factor is; if the symptoms re-arise I'll
>do more sniffing and try to find out more info.
>
>                 Tuc/TTSG Internet Services, Inc.