According to their announcement commemorating the event on the GNU.org site, September 27th, 1983 was the day that Richard Stallman first announced the GNU project to the public.

Today the open source software movement that GNU pioneered is vibrant, thriving, and global.   It counts among its members a multitude of individuals and project groups motivated by an amazing variety of values and goals.  Millions every day use open source software directly and innovation and competition from open source have spurred improvement even in projects that don’t share the open source philosophy.

At ISC we’re proud to be part of the open source movement that the GNU Project trailblazed.  We’d  like to salute our colleagues (and respected elder siblings) at GNU and congratulate them on their past 30 years of remarkable achievement.  Well done, GNU, and thank you for all that you have done to make the world a better place.

Response Rate Limiting (RRL)

A DNS DDoS attack works by forging queries that look like they came from the victim’s server, making it appear to be requesting a high volume of information. RRL enables server administrators to limit the rate at which their server will send replies to forged queries, thereby preventing it from contributing to the attack.

“Our users have been asking for RRL to be incorporated into BIND,” said Kannan Ayyar, President of Internet Systems Consortium, “and we recognize the important role it plays in DDoS mitigation. With DDoS attacks increasing in both number and severity, we felt it was important to integrate RRL into a supported release.”

“We have been testing RRL in limited release, and are now confident that it is ready for general use in BIND installations,” said Scott Mann, ISC’s VP of Engineering. “Third-party additions like RRL are possible because BIND is open source software. Now that it is fully implemented, we look forward to enhancing and building on RRL in future releases.”

For more information on RRL, visit the following links:

• DDoS Defense Module for BIND DNS – RRL (Webinar)
• A Quick Introduction to Response Rate Limiting
• Cache poisoning gets a second wind from RRL? Probably not.

Commercial support for BIND and additional RRL functionality, RRL Classifier, is available to The DNS Company subscription customers; visit The DNS Company’s BIND Solutions to learn more.

For questions, suggestions and discussions relevant to BIND, participate in our community mailing list, available at https://www.isc.org/community/mailing-list/.

BIND 9.9.4 is available for download at our Downloads page.

ISC acknowledges that RRL can increase the effectiveness of cache poisoning attacks and appreciates the detailed research that uncovered it.  This is, however, only one piece in the larger context of competing security concerns, and each operator will need to find their own balance of protection.

For those unfamiliar with it, RRL is designed to reduce the effectiveness of reflected denial of service (DoS) attacks which leverage DNS servers to amplify the attack.  DNS servers are frequently used as amplifying reflectors in DoS attacks because attackers can send a small UDP query with a forged source address to the DNS server and get it to respond with a much larger answer to the target of the DoS.  RRL reduces the effectiveness of these attacks by detecting when a large amount of similar traffic is being sent to a single target and suppressing responses.

This could, of course, potentially be used to create a different kind of DoS attack against a target if an attacker chooses to ask the same kinds of questions that the target is likely to ask.  If this were done against all of the servers authoritative for a zone then an attacker could potentially prevent the target from getting any answers at all for the zone.

In order to combat this risk, RRL was designed with a concept called “slip”.  Slip comes into play after RRL starts suppressing responses, and works by allowing a specified fraction of responses to “slip” through the suppression.  These responses contain none of the actual answer data, but do have the truncation (TC) bit set in the header of the response in order to tell the client to retry over TCP.  This enables legitimate clients to get answers via TCP, which has a lot more overhead than UDP but is not vulnerable to source address forgery.

ISC’s RRL implementation will debut in our upcoming 9.9.4 release.  Like the redbarn.org patches that preceded our implementation, we have chosen a default slip value of “2″, meaning that TC answers will be sent to the client/target 1 time in 2.  The other half of the time the queries will go unanswered.

It is these unanswered queries that create the increased opportunity for cache poisoning by giving an attacker a larger time window in which to get a forged reply with poison data to the victim.  The data that we’ve seen indicates that for a reasonably-configured resolver it takes, on average, more than sixteen hours of 100Mbps of forged answers in order to get the resolver to accept a poisoned answer.  During this time, the resolver is both being flooded with traffic and assumed to be unable to resolve the name in question.  Both of these are usually fairly visible events.  We have not seen any analysis of the expected time for a “stealthy” cache poisoning attack, but we expect it to be significantly longer.

The researchers who discovered this have recommended a slip value of “1″, sending a TC answer for every response that RRL decides needs to be suppressed.  This reduces the effectiveness of cache poisoning attacks while increasing the effectiveness of using DNS to amplify DoS attacks.

Note that this analysis only applies to queries and responses that are affected by RRL, while anything that causes the legitimate packets to be dropped, even simple traffic congestion, will benefit someone attempting cache poisoning.  Therefore, modifications to the behavior of RRL can have, at best, a limited effect in defending against cache poisoning. The best defense is for authoritative server operators to sign their zones with DNSSEC, and for resolver operators to validate responses.

The bottom line is that there is no clear “right” answer here.  Both concerns are valid and the mitigation for one increases the risk of the other.

We believe, based on what we know of the current state of the internet, that a slip value of “2″ is closer to the theoretical “sweet spot” in addressing both risks than a slip value of “1″ is, which is why we are keeping “2″ as our default.  Since RRL is not enabled by default even when it is compiled in to BIND, and the slip value is a configurable option, we believe that this provides the most useful default value while giving individual operators the freedom to choose the risk balance that they are comfortable with.

Finding the right risk balance also includes considering the effect that other features (e.g. DNSSEC) have on amplification potential and resistance to cache poisoning.

Those who are interested in learning more about how we got into this situation and where we ought to go from here may want to check out Paul Vixie’s blog post “On the Time Value of Security Features in DNS“.

A DNS DDoS attack works by forging queries that look like they came from the victim’s server, making it appear to be requesting a high volume of information. RRL enables server administrators to limit the rate at which their server will send replies to forged queries, thereby preventing it from contributing to the attack. The frequency of DNS DDoS attacks has been increasing, rising by 20% in Q2 of 2013. In an average attack 50 million packets per second are beamed at the victim. As attacks increase, RRL is the best defense available.

“Our users have been asking for RRL to be incorporated into BIND,” said Kannan Ayyar, President of Internet Systems Consortium, “and we recognize the important role it plays in DDoS mitigation. With DDoS attacks increasing in both number and severity, we felt it was important to integrate RRL into a supported release.”

“We have been testing RRL in limited release, and are now confident that it is ready for general use in BIND installations,” said Scott Mann, ISC’s VP of Engineering. “Third-party additions like RRL are possible because BIND is open source software. Now that it is fully implemented, we look forward to enhancing and building on RRL in future releases.”

For more information on RRL, visit the ISC Knowledgebase at https://kb.isc.org/article/AA-01000, or sign up for a webinar listed at our events page.

Commercial support for BIND and additional RRL functionality is available to DNSco subscription customers; visit DNSco’s BIND Solutions to learn more.

“Paul Vixie has been the driving force in Internet security innovation at ISC for many years,” said Kannan Ayyar, President of Internet Systems Consortium. “We are pleased that he will be taking these technologies forward and providing ongoing leadership in the security space. We look forward to his continued innovation there. Paul Vixie has been part of ISC for 18 years; with this new venture he has ended all of his involvement with ISC. We are grateful for his many contributions and share his excitement about Farsight.”

This is the second time this year that ISC has divested certain assets to a commercial for-profit entity. ISC is doing this to increase clarity and focus for its non-profit Internet mission, which includes running F-Root, providing hosting and Internet services for non- profits, researching and developing new ideas for the Internet and of course providing free world-class nameserver and DHCP software for the benefit of the Internet.

In April of this year, ISC launched DNSco (http://dns-co.com), a wholly-owned subsidiary delivering a full suite of commercial services for BIND and ISC DHCP. Farsight, unlike DNSco, is a privately held company that is independent of ISC.

“Farsight security technologies and service delivery capabilities are the result of years of research at ISC,” said Paul Vixie. “During that time we iterated on a range of business models and technology prototypes yielding best-in-class Internet security solutions that will help make the Internet a safer place.”

About ISC

Internet Systems Consortium is a non-profit 501(c)(3) public benefit corporation widely known for world-class Internet software engineering and network operations. ISC produces only open-source software with emphasis on Internet core technology, of which BIND and ISC DHCP are the two best-known examples. ISC’s Managed Open Source process ensures the quality of this software while keeping it open and available.

ISC operates high-reliability global networks of DNS root servers (F-root) and authoritative DNS servers (SNS@ISC) both for non-profit and for commercial enterprises. ISC is also very involved in ongoing Internet protocol and standards development, particularly in the areas of DNSSEC and IPv6. ISC is supported by donations from generous sponsors, by program membership fees, and by increasing revenues from for-profit subsidiaries. For program or donation information, please visit our website at http://isc.org.

About Farsight Security, Inc.

Farsight is a privately held Delaware corporation exclusively focused on the development of leading edge security solutions for ISPs, network and system security solution providers, governments, and medium to large commercial companies. Leveraging its superior telemetry data collection and processing capabilities, Farsight provides its clients with cloud-based, real-time network observability and reporting solutions.

Like ISC before it, Farsight is committed to sharing its security-related telemetry data with security industry partners and academic researchers at nominal, non-discriminatory subscription rates. In support of its mission as a clearing house for such data, Farsight invites network operators and commercial clients to provide additional telemetry data, which will increase the volume, quality and accuracy of its data provision services thus improving the overall safety of the Internet as a viable commercial marketplace.

Further information about it can be found at http://www.farsightsecurity.com

Internet Hall of Fame

It doesn’t appear, based on current observations, that this incident was due to malicious activity. However, ISC staff have identified multiple domains hosted by the registrar that are still having DNS queries for them directed to the wrong nameservers, as caches in recursive DNS resolvers all over the Internet have continued to hold incorrect records for them. This data distributed in error could persist in recursive servers, by some reports, for up to two days from the original incident before it times out, meaning that end users who rely on those servers might continue to be unable to reach the affected domains.

ISC Support staff have identified some steps that operators of recursive servers based on BIND can take to mitigate this issue by removing the bad data from your caches. The article is publicly available at KnowledgeBase, titled ‘How do I flush or delete incorrect records from my recursive server cache?’

It’s also worth pointing out that this case is not exactly the same as ‘cache poisoning,’ as the cached data was not introduced by a third party but published, at one time, as authoritative data.

ISC was recently involved in the troubleshooting and diagnosis of a DNSSEC-validation interoperability issue between BIND 9 and PowerDNS, where BIND is acting as a recursive server, and PowerDNS is authoritative. The end result was that BIND marked the PowerDNS server as not supporting EDNS0. Since DNSSEC requires EDNS0 support, the PowerDNS server was no longer considered capable of answering DNSSEC questions, and therefore, the BIND server was not able to resolve the DNSSEC-signed domains served by the PowerDNS authoritative servers.

EDNS0 is a DNS extension that has been part of the protocol since 1999, and was originally described in RFC 2671 (since obsoleted by RFC 6981). BIND 9, like most other recursive resolvers, supports this extension, and implements logic to work around DNS servers and network devices that do not understand EDNS0, or that do not behave properly if they do understand EDNS0. BIND must handle competing objectives when processing query responses from an authoritative server. If BIND sends an EDNS0-enabled query to a server and does not get a well-formed answer, it will try sending the query again, only this time without EDNS0. If an answer arrives, then the server is considered to work – but not with EDNS0.

The Issue in Detail

In the specific case described here, sometimes the PowerDNS authoritative server was returning an answer, but the response packet was malformed. Here is how the DNS packet flow transpired:

At this point, the BIND server has received a well-formed answer from the PowerDNS server – with EDNS0 disabled – so it marks the server as being able to answer, but this time without EDNS0.

Now, the main impact of marking a server as not being able to support EDNS0 is that DNSSEC requires EDNS0. This means that if all authoritative name servers for a domain are running PowerDNS and have the same error, then a DNSSEC-validating BIND 9 recursive server will not be able to resolve that DNSSEC-signed domain.

Further, this means that DNSSEC validation will no longer be possible for any of those servers’ DNSSEC-signed domains; if they are hosting thousands of domains secured by DNSSEC, all of them will become unresolvable.

Mitigation

From an administrative standpoint (i.e. what should the hapless DNS admin do about this), the solution is to either patch PowerDNS authoritative servers per the link above, or upgrade PowerDNS to version 3.3 RC. BIND administrators do not need to take any action unless they are running a validating recursive server that is already returning SERVFAILs due to marking the authoritative servers for the domains as EDNS0-incapable. In that situation, flushing the cache or restarting the server will restore normal service. It’s also particularly important in this case to be running a current version of BIND, as there have been bug fixes to cache management that are applicable to this situation – one in particular that ensures that an authoritative server’s EDNS0 capability (and other information) is refreshed after holding it for 30 minutes. Prior to this fix, authoritative server details were never discarded whilst the server was still being queried regularly.

Yet, while the actual fix for this situation is to upgrade the affected PowerDNS authoritative servers, it is theoretically possible for BIND to handle such partially-working servers better. One fix would be to always use EDNS0 when trying to get DNSSEC information from an authoritative server, rather than checking the server’s current status. This is what other resolvers (e.g. Unbound) do. The theory here is that if it works, then the user gets an answer; and if it does not work, it is no worse than not sending the query at all. Unfortunately, BIND 9′s current design does not make such information available when queries are sent. Changing the design to get around unusual situations like this would require a significant amount of engineering work, along with comprehensive functional and regression testing.

BIND could also try things such as requiring multiple failures, or shortening the time period before rechecking a server for EDNS0 support. These minor tweaks might help, and would not require major code changes. But they may also hurt in certain other cases, and again, would require significant retest effort. Ultimately, the BIND team decided not to make any code changes for this particular condition, since a fix exists for the PowerDNS server bug that is part of the problem, and this entirely mitigates the issue from a functional standpoint.

Looking Forward

Significant changes are pending in the EDNS0 handling code for BIND 9.10 which will make BIND behave better in its handling of EDNS0 in general. This should certainly improve the situation, but highlights one of the issues with changing protocols, and changing existing implementations. Postel’s law states: ”Be conservative in what you do, be liberal in what you accept from others.” BIND – and indeed all other production recursive resolvers – must be fairly liberal in handling implementation quirks.

The unfortunate truth is that, because of the way DNS works, it is often the resolver server’s operator who has to deal with the impact of broken authoritative servers. Even when the authoritative servers are responding correctly, the effects of an earlier problem can persist in cache and may require special administrator (or script) intervention. Until we discover reliable techniques to handle for the majority of situations, we’re stuck muddling through occasional breakages. This ISC Knowledge Base article may help those who need to handle those situations.

DNSco combines a business view and full-service solution with ISC’s world-renowned technical excellence. For over 15 years ISC has been the world’s leading expert in DNS and related technologies such as DHCP. ISC employees are versed in every aspect of these technologies, from protocol to implementation to operations to registries to policy. As the leader in these core Internet technologies and as the author and distributor of BIND and other core software, ISC has frequently been asked to provide business services to enterprises that recognize their dependence on DNS and DHCP.

“ISC realized that we could be more responsive to our commercial customers while remaining true to our original mission of improving and supporting Internet growth and stability,” said Kannan Ayyar, President of Internet Systems Consortium. “We are separating our globally respected public-benefit activities from our growing business activities. The mission of our new DNSco subsidiary is to provide credible, compatible, economical, stable, and expert support and services to businesses for which ISC’s software and technologies are mission-critical.”

DNSco is entirely owned by ISC, but will be managed separately. Its mission is different, its financials are separate, and it will be run as a commercial entity. But DNSco remains an ISC company, which means that expertise, staff, experience, and culture will move back and forth between the two companies. Separating commercial and public-benefit activities enables us do a better job of both. Moreover, profits from DNSco will help fund ISC’s ongoing public benefit mission.
DNSco’s products are subscriptions for major ISC software titles, training and consulting for the technologies involved, subscriber-only enhancements to the software, and software support related to their operation as well as advance notifications of security vulnerabilities.

In addition DNSco will release various closed-source software products, tools and services, including the newly launched back-end registry service. If DNS technologies are important to your business, or if they are your business, you will find DNSco products and services to be critical to your success.

Visit DNSco at http://www.dns-co.com

Media Contact:
Brian Reid
reid@isc.org
+1 650 423 1327

A significant component of the DDOS traffic targeted at Spamhaus is coming from a technique that has been known for years — a variety of reflection attack commonly known as a “DNS amplification attack.” By relying on the fact that an answer to a DNS query can be much larger than the query itself, attackers are able to both amplify the magnitude of the traffic directed against a DDOS victim and conceal the source of the attacking machines.

To accomplish this, the attacker sends a DNS query a few bytes in size to an open resolver, forging a “spoofed” source address for the query.  The open resolver, believing the spoofed source address, sends a response which can be hundreds of bytes in size to the machine it believes originated the request.  The end result is that the victim’s network connection is hit with several hundred bytes of information that were not requested.  They will be discarded when they reach the target machine, but not before exhausting a portion of the victim’s network bandwidth. And the traffic reaching the victim comes from the open resolver, not from the machine or machines used to initiate the attack.  Given a large list of open resolvers to reflect against, an attacker using a DNS amplification attack can hide the origin of their attack and magnify the amount of traffic they can direct at the victim by a factor of 40 or more.

DNS operators who operate open resolvers without taking precautions to prevent their abuse generally believe they are harming nobody, but as the Spamhaus DDOS proves, open resolvers can be effortlessly coopted by attackers and used in criminal attacks on third parties.

Beginning in 2005, ISC began publicly advocating for operators to stop operating open resolvers. In 2007 we changed the behavior of BIND, the world’s most popular nameserver software, so that open resolvers would no longer be the default. And in 2008 ISC CTO Joao Damas co-authored RFC 5358, “Preventing Use of Recursive Nameservers in Reflector Attacks.” For 8 years now we’ve been consistently leading on this issue as part of our mission to strengthen the DNS infrastructure, improve network security, and contribute to a stable and open internet. But there are still many open resolvers in operation. In order to avoid being pressed into service as an unwitting pawn in a criminal conspiracy, ISC strongly advises that DNS operators make use of the security features in BIND to enforce reasonable access permissions on their recursive resolvers.

At ISC, supporting the health of the Domain Name System and improving the security and stability of an open internet are core values and the biggest part of our public mission. If you would like to know more about us and our efforts, follow us on Twitter or Linked In, or get in touch via our contact page.

Next week we’ll have more to share about how current ISC development efforts are targeting reflection attacks and other network abuse to create a better internet for everybody.