DNS Resolution Failure - FORMERR

Tue May 5 23:08:54 UTC 2009

Eric Swenson wrote:
> I apologize for the multiple posts. I didn't think my post was making 
> it to the list since I never received my own post, but have been 
> receiving those of others.  And yes, I'm configured to see my own posts.
>
> A couple people have suggested I look at the trace output of bind to 
> see what server is sending the bad response.  I provide some of the 
> trace output below.  I certainly don't see anything amiss, and one of 
> the servers that appears to provoke the FORMERR seems to have 
> responded just fine.  Here is relevant output (with some stuff deleted 
> due to verbosity):
>
> 05-May-2009 10:49:14.943 dispatch 0x8144b90 response 0x81476b8 
> 192.228.79.201#53: attached to task 0x80ed240
> 05-May-2009 10:49:14.945 resquery 0x8152c70 (fctx 
> 0x812f170(imap.gmail.com/A) <http://imap.gmail.com/A%29>): sent
> 05-May-2009 10:49:14.945 resquery 0x8152c70 (fctx 
> 0x812f170(imap.gmail.com/A) <http://imap.gmail.com/A%29>): senddone
> 05-May-2009 10:49:14.945 dispatch 0x8149a70: got packet: requests 0, 
> buffers 2, recvs 1
> 05-May-2009 10:49:14.945 dispatch 0x8149a70: shutting down; detaching 
> from sock 0x81418f0, task 0x8141a20
> 05-May-2009 10:49:14.965 socket 0x8141460 192.228.79.201#53: packet 
> received correctly
> 05-May-2009 10:49:14.966 dispatch 0x8144b90: got packet: requests 1, 
> buffers 1, recvs 1
> 05-May-2009 10:49:14.966 dispatch 0x8144b90: got valid DNS message 
> header, /QR 1, id 47066
> 05-May-2009 10:49:14.966 resquery 0x8152c70 (fctx 
> 0x812f170(imap.gmail.com/A) <http://imap.gmail.com/A%29>): response
> 05-May-2009 10:49:14.967 received packet:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id:  47066
> ;; flags: qr rd ra ; QUESTION: 1, ANSWER: 3, AUTHORITY: 4, ADDITIONAL: 4
> ;; QUESTION SECTION:
> ;imap.gmail.com <http://imap.gmail.com>. IN A
>
> ;; ANSWER SECTION:
> imap.gmail.com <http://imap.gmail.com>. 241 IN CNAME 
> gmail-imap.l.google.com <http://gmail-imap.l.google.com>.
> gmail-imap.l.google.com <http://gmail-imap.l.google.com>. 241 IN A 
> 209.85.201.111
> gmail-imap.l.google.com <http://gmail-imap.l.google.com>. 241 IN A 
> 209.85.201.109
>
> ;; AUTHORITY SECTION:
> gmail.com <http://gmail.com>. 76384 IN NS ns4.google.com 
> <http://ns4.google.com>.
> gmail.com <http://gmail.com>. 76384 IN NS ns1.google.com 
> <http://ns1.google.com>.
> gmail.com <http://gmail.com>. 76384 IN NS ns2.google.com 
> <http://ns2.google.com>.
> gmail.com <http://gmail.com>. 76384 IN NS ns3.google.com 
> <http://ns3.google.com>.
>
> ;; ADDITIONAL SECTION:
> ns4.google.com <http://ns4.google.com>. 77136 IN A 216.239.38.10
> ns1.google.com <http://ns1.google.com>. 77136 IN A 216.239.32.10
> ns2.google.com <http://ns2.google.com>. 77136 IN A 216.239.34.10
> ns3.google.com <http://ns3.google.com>. 77136 IN A 216.239.36.10
This is a little suspect. Usually when you have a CNAME and A records in 
the Answer Section, the Authority Section contains NS records for the 
zone containing the A record (in this case, gmail-imap.l.google.com). 
Here, however, the Authority Section contains NS records for the zone 
containing the CNAME itself (imap.gmail.com). Odd. When I query the 
name, I get the l.google.com NS records in Authority.

I'm not sure, however, that this anomaly alone is sufficient to cause 
problems.
>
>
> 05-May-2009 10:49:14.967 fctx 0x812f170(imap.gmail.com/A' 
> <http://imap.gmail.com/A%27>): answer_response
> 05-May-2009 10:49:14.968 fctx 0x812f170(imap.gmail.com/A' 
> <http://imap.gmail.com/A%27>): noanswer_response
> 05-May-2009 10:49:14.968 fctx 0x812f170(imap.gmail.com/A' 
> <http://imap.gmail.com/A%27>): cancelquery
> 05-May-2009 10:49:14.968 dispatch 0x8144b90 response 0x81476b8 
> 192.228.79.201#53: detaching from task 0x80ed240
> 05-May-2009 10:49:14.968 dispatch 0x8144b90: detach: refcount 0
> 05-May-2009 10:49:14.968 fctx 0x812f170(imap.gmail.com/A' 
> <http://imap.gmail.com/A%27>): add_bad
> 05-May-2009 10:49:14.969 FORMERR resolving 'imap.gmail.com/A/IN 
> <http://imap.gmail.com/A/IN>': 192.228.79.201#53
>
> Does this trace output suggest what is going wrong?  -- Eric
>
> On Tue, May 5, 2009 at 9:53 AM, Eric Swenson <eric at swenson.org 
> <mailto:eric at swenson.org>> wrote:
>
>     I'm seeing lots of DNS resolution failures on my router (running
>     Utuntu 8.10, bind 9.3.4).  While most succeed, I get quite a few
>     FORMERR errors similar to:
>
>     May  4 20:25:25 localhost named[19579]: FORMERR resolving
>     'imap.gmail.com/A/IN <http://imap.gmail.com/A/IN>': 66.151.140.2#53
>     May  4 20:25:25 localhost named[19579]: FORMERR resolving
>     'imap.gmail.com/A/IN <http://imap.gmail.com/A/IN>': 192.168.3.1#53
>     May  4 20:25:25 localhost named[19579]: FORMERR resolving
>     'imap.gmail.com/A/IN <http://imap.gmail.com/A/IN>': 192.112.36.4#53
>     May  4 20:25:25 localhost named[19579]: FORMERR resolving
>     'imap.gmail.com/A/IN <http://imap.gmail.com/A/IN>': 128.63.2.53#53
>     May  4 20:25:25 localhost named[19579]: FORMERR resolving
>     'imap.gmail.com/A/IN <http://imap.gmail.com/A/IN>': 192.228.79.201#53
>     May  4 20:25:25 localhost named[19579]: FORMERR resolving
>     'imap.gmail.com/A/IN <http://imap.gmail.com/A/IN>': 192.36.148.17#53
>     May  4 20:25:25 localhost named[19579]: FORMERR resolving
>     'imap.gmail.com/A/IN <http://imap.gmail.com/A/IN>': 202.12.27.33#53
>     May  4 20:25:25 localhost named[19579]: FORMERR resolving
>     'imap.gmail.com/A/IN <http://imap.gmail.com/A/IN>': 192.33.4.12#53
>     May  4 20:25:25 localhost named[19579]: FORMERR resolving
>     'imap.gmail.com/A/IN <http://imap.gmail.com/A/IN>': 192.5.5.241#53
>     May  4 20:25:25 localhost named[19579]: FORMERR resolving
>     'imap.gmail.com/A/IN <http://imap.gmail.com/A/IN>': 192.58.128.30#53
>     May  4 20:25:25 localhost named[19579]: FORMERR resolving
>     'imap.gmail.com/A/IN <http://imap.gmail.com/A/IN>': 128.8.10.90#53
>     May  4 20:25:25 localhost named[19579]: FORMERR resolving
>     'imap.gmail.com/A/IN <http://imap.gmail.com/A/IN>': 198.41.0.4#53
>     May  4 20:25:25 localhost named[19579]: FORMERR resolving
>     'imap.gmail.com/A/IN <http://imap.gmail.com/A/IN>': 192.203.230.10#53
>     May  4 20:25:25 localhost named[19579]: FORMERR resolving
>     'imap.gmail.com/A/IN <http://imap.gmail.com/A/IN>': 193.0.14.129#53
>     May  4 20:25:25 localhost named[19579]: FORMERR resolving
>     'imap.gmail.com/A/IN <http://imap.gmail.com/A/IN>': 199.7.83.42#53
>
Most of those are root servers, I'd highly doubt that you're getting 
FORMERR responses directly back from them. In fact, I tried that same 
query myself, and didn't get a FORMERR. My guess is that something is 
munging the packets before they get back to you.
>
>
>     I'm running an iptables firewall on this box, which is connected
>     to the internet via a wireless access point on my roof with a link
>     to my ISP.  As a result of the above FORMERRs, clients on my lan
>     are unable to resolve addresses -- in the above
>     case, imap.gmail.com <http://imap.gmail.com>, and therefore are
>     unable to access mail.  Upon the recommendations of someone
>     familiar with the relevant technologies, I've updated my DNS
>     (named.conf) to set the edns-udp-size 500 option.  This had no
>     effect.  
>
>     If I use dig to resolve imap.gmail.com
>     <http://imap.gmail.com> manually, by specifying any of the
>     above-mentioned DNS servers, everything works fine.  Also, when
>     clients within my network fail to have imap.gmail.com
>     <http://imap.gmail.com> resolve, I can "fix" things for a short
>     while, by simply issuing the following:
>
>     nslookup
>     set querytype=ns
>     gmail.com <http://gmail.com>.
>     lserver <whatever-the-ns-server-is-for-gmail.com
>     <http://whatever-the-ns-server-is-for-gmail.com>>
>     set querytype=a
>     imap.gmail.com <http://imap.gmail.com>
>
>     Once I've done the above, my DNS server caches the A record for
>     imap.gmail.com <http://imap.gmail.com> and happily hands it out
>     until the cache time is exceeded, when I'm back getting FORMERRs
>     and failing to resolve imap.gmail.com <http://imap.gmail.com>.
>
>     There are other addresses than imap.gmail.com
>     <http://imap.gmail.com> that cannot be resolved due to FORMERRs,
>     but this domain name is the most prevalent, and most annoying,
>     since it prevents users within my network from getting mail.
>
>     Since I can force my DNS to resolve these addresses by issuing the
>     above queries, I'm wondering if the problem is due to having the
>     following in my named.conf:
>
>      forwarders {
>              192.168.3.1;
>              66.151.140.2;
>      };
>
>     My ISP provides the above two DNS servers and I have mine
>     delegating to theirs. 
>
Don't know what you mean by that. Delegation is from a parent *zone* to 
a child *zone*. It's a relationship between records in the DNS database. 
Did you mean "forwarding" perhaps? "Delegating" seems to be the wrong word.

Note that if you don't specify "forward only", the default forwarding 
mode is "forward first", which means named tries the forwarders, and if 
that doesn't work, then it falls back to iterative resolution, starting 
back up at the root zone, if necessary. An absence of "forward only" 
would explain why you're seeing queries directly to the root servers in 
addition to your forwarders. I would suggest turning on "forward only" 
until you get this sorted out -- the root servers don't need more "junk 
traffic" than they're already getting, and since you're not accepting, 
and not caching, those allegedly-FORMERR responses that you're seeing, 
you'll keep going back to the roots over and over again.
>
>     Perhaps one of these two DNS servers (or any that they forward to)
>     is having problems (perhaps no EDNS0 support?), which causes the
>     FORMERRs to be reported by my DNS server.
>
>     I haven't yet tried removing the forwarders.  I figured this was
>     not the issue because the FORMERR log messages suggest (to me)
>     that my DNS is trying to contact the root servers itself (and not
>     relying on the downstream DNS servers to do so).  
>
>     Does anyone have ideas about what is going on?
>

Time to break out the packet sniffer and see what's really going on.

                           - Kevin