DNSSEC error resolving gpo.gov ?

Tue Apr 4 05:07:12 UTC 2023

Please tell them to forward my dig output to the DNS vendor.  The header content does NOT match the rest of the packet.  Just because some resolvers “work” with malformed packets it doesn’t mean that the servers are not broken.

The headers say that there are 3 records in the answer section and 1 in the additional section when there is only 1 record present.  That is the definition of a malformed response.

Mark

> On 4 Apr 2023, at 12:41, Alexandra Yang <drayales at gmail.com> wrote:
> 
> Hi Mark,
> 
> We just heard back from the gpo.gov nameserver, see below. Looks like they think the cause of the problem is BIND not resend query using TCP which is required since DNS Flag day 2020 ? Thanks again for your help!
> 
> " 
>     • We also analyzed DNSSEC for federalregister.gov against the following major DNS Servers and they have come back as fully validated.
>         • Google (8.8.8.8)
>         • Cloudflare (1.1.1.1)
>         • OpenDNS (208.67.220.220)
>     • During our extensive testing, we were able to reproduce the issue and found out  that DNS requests could fail if remote DNS Servers ( forwarders ) are not compliant with the requirements for the DNS environments since DNS Flag Day 2020 ( https://www.dnsflagday.net/2020/ )
>         • Most importantly: Resolvers MUST resend queries over TCP if they receive a truncated UDP response (with TC=1 set)!
>  Our recommendation is please review the requirements for the DNS environments since DNS Flag Day 2020 ( https://www.dnsflagday.net/2020/ ) and make sure the local forwarders are compliant.
>  "
> 
> 
> On Wed, Mar 15, 2023 at 6:01 PM Mark Andrews <marka at isc.org> wrote:
> 
> 
> > On 15 Mar 2023, at 16:49, Mark Andrews <marka at isc.org> wrote:
> > 
> > 
> > 
> >> On 15 Mar 2023, at 15:42, Tim Maestas <tmaestas95 at gmail.com> wrote:
> >> 
> >> Named should be sending queries with DO=1 and it should be getting back signed responses.  I suspect that you will need to run packet captures of the traffic to and from 162.140.15.100 and 162.140.254.200 port 53 from the nameserver.  Either signed responses will cease or DNSSEC requests will cease.  In either  case having the traffic around the transition should help to determine what is happening.
> >> 
> >> I've found that, after a fresh restart of named, if I query for "federalregister.gov A" I get a good AD response, and then subsequent queries for "www.federalregister.gov" are successful as well.  If however after a restart of named I begin with a query for www.federalregister.gov A then I get servfail, and subsequent queries for federealregister.gov servfail as well.  Here is the tcpdump from the 2nd (failed) case of an initial query for www.federalregister.gov:
> >> 
> >> 
> >> reading from file dns.cap, link-type EN10MB (Ethernet), snapshot length 262144
> >> 04:30:01.114458 IP (tos 0x0, ttl 64, id 35832, offset 0, flags [none], proto UDP (17), length 92)
> >>    10.0.0.159.43263 > 162.140.254.200.53: [udp sum ok] 15013 [1au] A? www.federalregister.gov. ar: . OPT UDPsize=512 DO [COOKIE 352538a87bde87a5] (64)
> >> 04:30:01.204863 IP (tos 0x0, ttl 229, id 4936, offset 0, flags [DF], proto UDP (17), length 80)
> >>    162.140.254.200.53 > 10.0.0.159.43263: [udp sum ok] 15013*-| q: A? www.federalregister.gov. 3/0/1 . OPT UDPsize=4096 DO [|domain]
> > 
> > This is a malformed DNS response.  It looks like the server tried to send a truncated response with an OPT record but failed to correctly update the answer count field to zero.  
> > 
> > % dig www.federalregister.gov @162.140.15.100 +dnssec +bufsize=512 +ignore +qr +norec
> > 
> > ; <<>> DiG 9.19.11-dev <<>> www. @162.140.15.100 +dnssec +bufsize=512 +ignore +qr +norec
> > ;; global options: +cmd
> > ;; Sending:
> > ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57919
> > ;; flags: ad; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
> > 
> > ;; OPT PSEUDOSECTION:
> > ; EDNS: version: 0, flags: do; udp: 512
> > ; COOKIE: 4a67cc813cfe03eb
> > ;; QUESTION SECTION:
> > ;www.federalregister.gov. IN A
> > 
> > ;; QUERY SIZE: 64
> > 
> > ;; Warning: Message parser reports malformed message packet.
> > ;; Got answer:
> > ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57919
> > ;; flags: qr aa tc; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
> > 
> > ;; QUESTION SECTION:
> > ;www.federalregister.gov. IN A
> > 
> > ;; ANSWER SECTION:
> > . 32768 CLASS4096 OPT 
> > ;; Query time: 271 msec
> > ;; SERVER: 162.140.15.100#53(162.140.15.100) (UDP)
> > ;; WHEN: Wed Mar 15 16:30:22 AEDT 2023
> > ;; MSG SIZE  rcvd: 52
> 
> 
> What I should have said about this is that this tells named that the server DOES NOT SUPPORT EDNS and hence DNSSEC as there is no OPT record in the ADDITIONAL section in a response where the port, query id and question match the request.
> 
> Now I have reported this via the web site but whether they take it seriously or not only time will tell.  It’s very hit and miss trying to report broken DNS servers.  The automated response is below.
> 
> Mark
> 
> Your request (42885) has been received and is being reviewed by our support staff.
> 
> Thank you for contacting the Office of the Federal Register (OFR).
> 
> The OFR publishes on behalf of more than 300 Federal entities and can only accept documents from Federal agencies. We have no involvement with, or control over, their policies or programs. The OFR is responsible for the accuracy of the content of the Federal Register and CFR, which must reflect what the agency submitted for publication in the Federal Register. Federal regulations do not permit OFR staff to provide assistance for requests asking the OFR to:
> - evaluate or interpret material published in the Federal Register;
> - provide legal analysis of material published in the Federal Register;
> - explain or provide compliance guidance of material published in the Federal Register.
> In addition, we are not permitted to perform research or make detailed suggestions to locate material published on specific topics. We will not respond to those emails.
> 
> If you have a question about our office or one of our publications, we will respond within 10 business days. Otherwise, use the information below to find an appropriate contact.
> 
> 1. For problems with ecfr.gov:
> - use this help system
> - to correct a content error that was incorrect in the Federal Register rule, contact the agency that maintains the regulations
> 
> 2. To find regulations or requirements or material on specific topics:
> - use the terms you have included in this message in the search feature at https://www.ecfr.gov/search, to search across all CFR titles,
> - use the terms you have included in this message in the search features at https://www.federalregister.gov, or
> - search the U.S. Government’s official web portal, https://www.usa.gov <https://www.usa.gov/>
> 
> 3. To ask questions about an agency document:
> - contact the issuing agency. To find contact information:
> - search the agency's regulations (using the search feature at https://www.ecfr.gov/search)
> - look in either ADDRESSES or FOR FURTHER INFORMATION CONTACT sections of the relevant Federal Register document
> - visit the agency’s website to find contact information
> - search the U.S. Government’s official web portal, https://www.usa.gov <https://www.usa.gov/>
> 
> 4. For help with research, visit a local Federal Depository Library (https://www.fdlp.gov/about-the-fdlp/federal-depository-libraries).
> 
> ecfr.gov does not link to outside interests and we do not accept or respond to solicitations.
> ———
> Mark Andrews
> Mar 15, 2023, 1:47 AM EDT
> 
> Your DNS servers are sending malformed DNS responses. The server attempted to send a truncated DNS response but the answer count was NOT set to zero in the DNS header causing the OPT record to appear in the answer section rather that the additional section. You will need to contact your DNS vendor for a fix. This is causing DNS resolution failures in some circumstances. 
> 
> % dig www.federalregister.gov @162.140.15.100 +dnssec +bufsize=512 +ignore +qr +norec
> ; <<>> DiG 9.19.11-dev <<>> www.federalregister.gov @162.140.15.100 +dnssec +bufsize=512 +ignore +qr +norec
> ;; global options: +cmd
> ;; Sending:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57844
> ;; flags: ad; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
> ;; OPT PSEUDOSECTION:
> ; EDNS: version: 0, flags: do; udp: 512
> ; COOKIE: ba894ad3a75d0394
> ;; QUESTION SECTION:
> ;www.federalregister.gov. IN A
> ;; QUERY SIZE: 64
> ;; Warning: Message parser reports malformed message packet.
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57844
> ;; flags: qr aa tc; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
> ;; QUESTION SECTION:
> ;www.federalregister.gov. IN A
> ;; ANSWER SECTION:
> . 32768 CLASS4096 OPT
> ;; Query time: 329 msec
> ;; SERVER: 162.140.15.100#53(162.140.15.100) (UDP)
> ;; WHEN: Wed Mar 15 16:41:53 AEDT 2023
> ;; MSG SIZE rcvd: 52
> -- 
> Mark Andrews, ISC
> 1 Seymour St., Dundas Valley, NSW 2117, Australia
> PHONE: +61 2 9871 4742              INTERNET: marka at isc.org
> 
> -- 
> Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list
> 
> ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.
> 
> 
> bind-users mailing list
> bind-users at lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742              INTERNET: marka at isc.org