EDNS issue with bind 9.11 and NetScaler 11.0

[Mark Andrews]

In message <12165D85-6CC2-4C9E-9001-45B20D6D6676 at nau.edu>, Mathew Ian Eis writes:
> Hi BIND,
>
> We are running BIND behind a Citrix NetScaler (v 11.0) load balancer, and
> recently had a report that BIND 9.11 is unable to resolve names from our
> public nameservers.
>
> The issue can be easily reproduced with the BIND 9.11 client, e.g.: $ dig
> nau.edu @a.ns.nau.edu (will return status: FORMERR).

BIND 9.11 and BIND 9.10 (if enabled at compile time - on by default
for Windows) implements DNS Cookies (RFC 7873) which is anti spoofing
technology.  It does this by sending a EDNS Cookie option with the
request.

> $ dig +noedns nau.edu @a.ns.nau.edu on the other hand works.

dig +nocookie is a slightly less drastic way of removing the cookie
option from the request.

> The report passed on to us was secondhand, but ISC reportedly thinks that
> EDNS support is broken on the load balancer I think that conversation
> must have been off list as this report was the first wed heard about it.

It is.  The correct response is to ignore unknown EDNS options not
return FORMERR (RFC 6891).  The box also fails to perform EDNS
version negotiation correctly.  See the report below.

The IETF looked at bumping the EDNS version number when RFC 6891
was released but nameservers either already ignored unknown EDNS
options, just copied the EDNS option or returned FORMERR regardless
of the EDNS version number which made bumping the version number
pointless when we clarified the behavior.

Add to that all the firewalls that blocked EDNS version != 0 queries
it was decided to clarify without bumping the version number was
the least worst plan of action.

This is a bigger issue for you as your zones are signed and named
falls back to plain DNS on FORMERR.  This breaks DNSSEC validation.

You should also have email in your mail box from me.  I sent it last
night.

ISC has a EDNS compliance tester as well 2+ years of data on EDNS
compliance trends at https://ednscomp.isc.org.

We are also working to release the following I-D of which EDNS
compliance is a subset of the issues being addressed.
https://tools.ietf.org/html/draft-ietf-dnsop-no-response-issue-06

Microsoft are working on fixing their servers.  All the Azure servers
are currently broken.

Checkpoint are removing the blocks on queries with EDNS version != 0
and those with EDNS flags set other than DO.

AWS (Route 53) have partially fixed their servers.  They no longer
drop EDNS version 1 queries.  They still however need to EDNS version
negotiation.

Fixing these bugs is about 10 minutes work for the developer.

> We are working with Citrix to try and resolve the issue, but I wanted to
> ask - has anyone else seen this, and if so, how did you resolve it?

Can you also ask Citrix to issue a CVE against the existing products.

> Also, Evan, if you can provide any more info on the issue that we can
> pass it on to Citrix, that would be much appreciated.
>
> Thanks in advance,
>
> Mathew Eis
> Northern Arizona University
> Information Technology Services

Checking: 'nau.edu' as at 2016-12-20T22:48:22Z

nau.edu @134.114.93.82 (a.ns.nau.edu.): edns=ok edns1=formerr,badversion edns at 512=ok ednsopt=formerr,echoed,nosoa edns1opt=formerr,badversion,echoed do=ok ednsflags=ok docookie=formerr,nosoa,echoed edns at 512tcp=ok optlist=formerr,nosoa,subnet
nau.edu @134.114.93.83 (b.ns.nau.edu.): edns=ok edns1=formerr,badversion edns at 512=ok ednsopt=formerr,echoed,nosoa edns1opt=formerr,badversion,echoed do=ok ednsflags=ok docookie=formerr,nosoa,echoed edns at 512tcp=ok optlist=formerr,nosoa,subnet
The Following Tests Failed

Warning: test failures may indicate that some DNS clients cannot resolve the zone or will get a unintended answer or resolution will be slower than necessary.

Warning: failure to address issues identified here may make future DNS extensions that you want to use ineffective. In particular echoing back unknown EDNS options and unknown EDNS flags will break future signaling between DNS client and DNS server. We already have examples of this were you cannot depend on the AD flag bit meaning anything in replies because too many DNS servers just echo it back. Similarly the EDNS Client Subnet (ECS) option cannot just be sent to everyone in part because of servers just echoing it back.

EDNS - Unknown Version Handling (edns1)

dig +nocookie +norec +noad +edns=1 +noednsneg soa zone @server
expect: BADVERS
expect: OPT record with version set to 0
expect: not to see SOA
See RFC6891, 6.1.3. OPT Record TTL Field Use

EDNS - Unknown Option Handling (ednsopt)

dig +nocookie +norec +noad +ednsopt=100 soa zone @server
expect: SOA
expect: NOERROR
expect: OPT record with version set to 0
expect: that the option will not be present in response
See RFC6891, 6.1.2 Wire Format

EDNS - Unknown Version with Unknown Option Handling (edns1opt)

dig +nocookie +norec +noad +edns=1 +noednsneg +ednsopt=100 soa zone @server
expect: BADVERS
expect: OPT record with version set to 0
expect: not to see SOA
expect: that the option will not be present in response
See RFC6891

EDNS - DNSSEC with DNS COOKIE Option (docookie)

This is the style of the initial query that BIND 9.11.0 and BIND 9.10.4 Windows onwards send.

dig +cookie +norec +noad +dnssec soa zone @server
expect: SOA
expect: NOERROR
expect: OPT record with version set to 0
expect: DO flag in response if RRSIG is present in response
See RFC3225, RFC6891, and RFC7873.

EDNS - Supported Options Probe (optlist)

dig +edns +noad +norec +nsid +subnet=0.0.0.0/0 +expire +cookie -q zone @server
expect: NOERROR
expect: OPT record with version set to 0
See RFC6891

Codes

ok - test passed.
subnet - EDNS Client Subnet supported [RFC7871].
nosoa - SOA record not found when expected.
echoed - EDNS option echoed back.
formerr - rcode FORMERR returned.
badversion - expected EDNS version not found.
To retrieve this report in the future: https://ednscomp.isc.org/ednscomp/e6873acb4d



-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: marka at isc.org