Happy holidays from ISC!
ISC is fortunate to have staff members in so many different countries around the world: our software development benefits from all the different perspectives - and we benefit personally!
Read post
A number of DNS software and service providers have announced that we will all cease implementing DNS resolver workarounds to accommodate DNS authoritative systems that don’t follow the Extensions to DNS (EDNS) protocol. Each vendor has pledged to roll out this change in some version of their software by the “Flag Day.”
Changes coming in BIND
BIND 9.13.4 and later versions remove the EDNS workarounds, and BIND 9.14.0, which will be posted soon after Flag Day, will also remove those workarounds. This change will NOT be back-ported to the 9.11 or earlier branches of BIND. This is because of our policy of not making unnecessary changes to our stable extended support release.
The BIND 9 authoritative server has always been EDNS-compliant (since EDNS was specified).
Non-compliant domains may become unavailable
Domains served by DNS servers that are not compliant with the standard will not function reliably when queried by resolvers that have been updated to the post-Flag Day version, and may become unavailable via those updated resolvers.
If your company’s DNS zones are served by non-compliant servers, your online presence will slowly degrade or disappear as ISPs and other organizations update their resolvers. When you update your own internal DNS resolvers to versions that don’t implement workarounds, some sites and email servers may become unreachable.
Test your domains
Operators of DNS authoritative systems should check their own systems at https://www.dnsflagday.net/ to ensure they are EDNS-compliant. Most common failures are due to firewalls blocking EDNS traffic, or older DNS servers that need to be upgraded.
If you are running any supported version of BIND 9, you are compliant already. If the test site isn’t working (there has been quite a spike in traffic recently!), you can also run the tests manually, using DIG. See instructions on that at the bottom of this blog. Keep in mind that if you are scanning for potential problems with your firewalls, you need to run the test across that perimeter.
If you run the tests from ednscomp.isc.org, and you get failures due to timeouts, consider that possibly your RRL settings on your authoritative servers may be rate-limiting the queries coming from the test tool. ednscomp.isc.org is sending dig queries from 149.20.1.48 and 2001:4f8:1:f::48. Temporarily whitelisting those addresses in your RRL settings could fix the problem. We have also had to protect ednscomp.isc.org from mass scanning, so if you are running many queries, you will also be rate limited by ednscomp.isc.org.
Operators of resolvers should consult the table below, or their vendor, to determine whether the software they are running has EDNS workarounds or not. If/when workarounds are removed, resolver operators need to be on the lookout for reports that some websites have become unreachable, and should consider EDNS incompatibility to be one of the potential causes when troubleshooting. If that turns out to be the case, the solution is to notify the problem site so that they can remediate the problem.
At ISC, we have been reaching out to operators of non-compliant sites for years, urging them to update. Although there are still a few non-compliant sites that will cause problems for the rest of us, the only long-term solution is for the operators of those sites to resolve their compliance problems.
Open source DNS software plans
The organizations who have agreed to update their software or systems include:
| Organization | Resolver Version for Flag Day | Notes |
|---|---|---|
| ISC | BIND 9.14 (stable) to be released at the end of January 2019 – removes resolver workarounds for servers that misbehave when queried with EDNS | BIND 9's authoritative systems are compliant with EDNS already. The change in 9.14 removes the workarounds from the BIND resolver. |
| CZ NIC | Knot Resolver – newer than the others – already written without most of the workarounds for misbehaving servers. 3.3.0 (soon to be released) has some minor changes | In general, anyone already running the latest version (3.2.0) should not notice any significant differences when upgrading to 3.3.0. |
| NLNET Labs | Unbound versions released after 1st February 2019 (1.8.4, 1.9.0 and newer) will no longer retry without EDNS when no response is received | Unbound will still accept answers without EDNS, and will still send a query without EDNS when it receives a FORMERR or NOTIMPL answer. (This change will only affect queries that result in a time-out because of EDNS in the query.) |
| PowerDNS | PowerDNS recursor 4.2 (to be released soon) will be the first one to no longer accommodate non-compliance | On the authoritative side, PowerDNS 4.1 is fully compliant; 4.0 has some corner cases that ednscomp notices but that are not a problem in practice – disabling caching removes those edge cases. |
Why make this change now?
Extension Mechanisms for DNS were specified in 1999, with a minor update in 2013, establishing the “rules of the road” for responding to queries with EDNS options or flags. Despite this, some implementations continue to violate the rules. DNS software developers have tried to solve the problems with the interoperability of the DNS protocol and especially its EDNS extension (RFC 6891 standard) by various workarounds for non-standard behaviors. This is not unlike the way a driver with the right-of-way might hesitate at an intersection before proceeding if there were another driver in the intersection behaving erratically. These workarounds excessively complicate DNS software and are now also negatively impacting the DNS as a whole.
The most obvious problems caused by these workarounds are slower responses to DNS queries and the difficulty of deploying new DNS protocol features. Some of these new features (e.g. DNS Cookies) would help reduce DDoS attacks based on DNS protocol abuse.
At ISC we have been probing the DNS for years, looking at EDNS compatibility. We have seen significant improvement over the past four years, and we think this is about as good as we will get as long as we have workarounds in place. Below is one of a number of charts available at https://ednscomp.isc.org/compliance/summary.html. If you are interested, look at some of the other charts which show improvements in EDNS awareness and EDNS compatibility over the years. Some of the EDNS compatibility problems noted are more serious than others.
Our goal is a reliable and properly functioning DNS that cannot be easily attacked.
Resources and References
Earlier blog on EDNS fallback behavior
Presentation on DNS Flag Day on UKNOF, January 15th, 2019
KB article – DNS Flag Day, How Will it Impact You?
KB article – DNS Flag Day, Notes for Authoritative servers
KB article – EDNS compatibility tests and status codes
Checking EDNS Compatibility with DIG
The ednscomp.isc.org site uses a modified version of DIG to run multiple tests in series. This test version is available for download at https://gitlab.isc.org/isc-projects/DNS-Compliance-Testing.
You can also use the dig included in the BIND distribution and run multiple queries yourself to check a suspicious domain. For instructions on how to do this, see this Knowledgebase article.