- By Adib Behjat on June 20, 2013
Last night, TechCrunch reported that LinkedIn and Fidelity.com faced an outage due to a DNS error. ISC staff and colleagues observed that the error was caused due to the changing of nameserver information at the registry; leading to DNS queries to be directed to nameservers that did not correctly answer those queries. Suzanne Woolf, ISC’s Director of Strategic Partnerships, points out “the problem was apparently aggravated by a long TTL on the incorrect nameserver records, causing the bad information to persist in resolver caches and the misdirection of queries to continue far longer than such information is usually held”. Eric Ziegast, member of the security products group for ISC, also observed via DNSDB that over 40 domains have been affected as well.
It doesn’t appear, based on current observations, that this incident was due to malicious activity. However, ISC staff have identified multiple domains hosted by the registrar that are still having DNS queries for them directed to the wrong nameservers, as caches in recursive DNS resolvers all over the Internet have continued to hold incorrect records for them. This data distributed in error could persist in recursive servers, by some reports, for up to two days from the original incident before it times out, meaning that end users who rely on those servers might continue to be unable to reach the affected domains.
ISC Support staff have identified some steps that operators of recursive servers based on BIND can take to mitigate this issue by removing the bad data from your caches. The article is publicly available at KnowledgeBase, titled ‘How do I flush or delete incorrect records from my recursive server cache?’
It’s also worth pointing out that this case is not exactly the same as ‘cache poisoning,’ as the cached data was not introduced by a third party but published, at one time, as authoritative data.
- By Jeff Wright on June 20, 2013
Unexpected DNSSEC validation failures
ISC was recently involved in the troubleshooting and diagnosis of a DNSSEC-validation interoperability issue between BIND 9 and PowerDNS, where BIND is acting as a recursive server, and PowerDNS is authoritative. The end result was that BIND marked the PowerDNS server as not supporting EDNS0. Since DNSSEC requires EDNS0 support, the PowerDNS server was no longer considered capable of answering DNSSEC questions, and therefore, the BIND server was not able to resolve the DNSSEC-signed domains served by the PowerDNS authoritative servers.
EDNS0 is a DNS extension that has been part of the protocol since 1999, and was originally described in RFC 2671 (since obsoleted by RFC 6981). BIND 9, like most other recursive resolvers, supports this extension, and implements logic to work around DNS servers and network devices that do not understand EDNS0, or that do not behave properly if they do understand EDNS0. BIND must handle competing objectives when processing query responses from an authoritative server. If BIND sends an EDNS0-enabled query to a server and does not get a well-formed answer, it will try sending the query again, only this time without EDNS0. If an answer arrives, then the server is considered to work – but not with EDNS0.
The Issue in Detail
In the specific case described here, sometimes the PowerDNS authoritative server was returning an answer, but the response packet was malformed. Here is how the DNS packet flow transpired:
At this point, the BIND server has received a well-formed answer from the PowerDNS server – with EDNS0 disabled – so it marks the server as being able to answer, but this time without EDNS0.
Now, the main impact of marking a server as not being able to support EDNS0 is that DNSSEC requires EDNS0. This means that if all authoritative name servers for a domain are running PowerDNS and have the same error, then a DNSSEC-validating BIND 9 recursive server will not be able to resolve that DNSSEC-signed domain.
Further, this means that DNSSEC validation will no longer be possible for any of those servers’ DNSSEC-signed domains; if they are hosting thousands of domains secured by DNSSEC, all of them will become unresolvable.
From an administrative standpoint (i.e. what should the hapless DNS admin do about this), the solution is to either patch PowerDNS authoritative servers per the link above, or upgrade PowerDNS to version 3.3 RC. BIND administrators do not need to take any action unless they are running a validating recursive server that is already returning SERVFAILs due to marking the authoritative servers for the domains as EDNS0-incapable. In that situation, flushing the cache or restarting the server will restore normal service. It’s also particularly important in this case to be running a current version of BIND, as there have been bug fixes to cache management that are applicable to this situation – one in particular that ensures that an authoritative server’s EDNS0 capability (and other information) is refreshed after holding it for 30 minutes. Prior to this fix, authoritative server details were never discarded whilst the server was still being queried regularly.
Yet, while the actual fix for this situation is to upgrade the affected PowerDNS authoritative servers, it is theoretically possible for BIND to handle such partially-working servers better. One fix would be to always use EDNS0 when trying to get DNSSEC information from an authoritative server, rather than checking the server’s current status. This is what other resolvers (e.g. Unbound) do. The theory here is that if it works, then the user gets an answer; and if it does not work, it is no worse than not sending the query at all. Unfortunately, BIND 9′s current design does not make such information available when queries are sent. Changing the design to get around unusual situations like this would require a significant amount of engineering work, along with comprehensive functional and regression testing.
BIND could also try things such as requiring multiple failures, or shortening the time period before rechecking a server for EDNS0 support. These minor tweaks might help, and would not require major code changes. But they may also hurt in certain other cases, and again, would require significant retest effort. Ultimately, the BIND team decided not to make any code changes for this particular condition, since a fix exists for the PowerDNS server bug that is part of the problem, and this entirely mitigates the issue from a functional standpoint.
Significant changes are pending in the EDNS0 handling code for BIND 9.10 which will make BIND behave better in its handling of EDNS0 in general. This should certainly improve the situation, but highlights one of the issues with changing protocols, and changing existing implementations. Postel’s law states: ”Be conservative in what you do, be liberal in what you accept from others.” BIND – and indeed all other production recursive resolvers – must be fairly liberal in handling implementation quirks.
The unfortunate truth is that, because of the way DNS works, it is often the resolver server’s operator who has to deal with the impact of broken authoritative servers. Even when the authoritative servers are responding correctly, the effects of an earlier problem can persist in cache and may require special administrator (or script) intervention. Until we discover reliable techniques to handle for the majority of situations, we’re stuck muddling through occasional breakages. This ISC Knowledge Base article may help those who need to handle those situations.
- By Adib Behjat on April 9, 2013
Beijing, China – 9 April 2013. Internet Systems Consortium (ISC) announces the creation of its first commercial subsidiary, DNSco.
￼DNSco combines a business view and full-service solution with ISC’s world-renowned technical excellence. For over 15 years ISC has been the world’s leading expert in DNS and related technologies such as DHCP. ISC employees are versed in every aspect of these technologies, from protocol to implementation to operations to registries to policy. As the leader in these core Internet technologies and as the author and distributor of BIND and other core software, ISC has frequently been asked to provide business services to enterprises that recognize their dependence on DNS and DHCP.
“ISC realized that we could be more responsive to our commercial customers while remaining true to our original mission of improving and supporting Internet growth and stability,” said Kannan Ayyar, President of Internet Systems Consortium. “We are separating our globally respected public-benefit activities from our growing business activities. The mission of our new DNSco subsidiary is to provide credible, compatible, economical, stable, and expert support and services to businesses for which ISC’s software and technologies are mission-critical.”
DNSco is entirely owned by ISC, but will be managed separately. Its mission is different, its financials are separate, and it will be run as a commercial entity. But DNSco remains an ISC company, which means that expertise, staff, experience, and culture will move back and forth between the two companies. Separating commercial and public-benefit activities enables us do a better job of both. Moreover, profits from DNSco will help fund ISC’s ongoing public benefit mission.
DNSco’s products are subscriptions for major ISC software titles, training and consulting for the technologies involved, subscriber-only enhancements to the software, and software support related to their operation as well as advance notifications of security vulnerabilities.
In addition DNSco will release various closed-source software products, tools and services, including the newly launched back-end registry service. If DNS technologies are important to your business, or if they are your business, you will find DNSco products and services to be critical to your success.
Visit DNSco at http://www.dns-co.com
+1 650 423 1327
- By Michael McNally on March 28, 2013
It has been a very eventful week in the field of DNS operations. In addition to the BIND vulnerability disclosed by ISC this week, the DNS world has been buzzing with news about “the biggest Distributed Denial of Service attack to date”, directed against Spamhaus by parties critical of their decision to list Cyberbunker as a spam source. As an industry leader in the field of DNS software, ISC sees the Spamhaus DDOS as a perfect opportunity to remind DNS operators why it is important to not operate an “open” recursive resolver, a policy recommendation we have been making since 2005.
A significant component of the DDOS traffic targeted at Spamhaus is coming from a technique that has been known for years — a variety of reflection attack commonly known as a “DNS amplification attack.” By relying on the fact that an answer to a DNS query can be much larger than the query itself, attackers are able to both amplify the magnitude of the traffic directed against a DDOS victim and conceal the source of the attacking machines.
To accomplish this, the attacker sends a DNS query a few bytes in size to an open resolver, forging a “spoofed” source address for the query. The open resolver, believing the spoofed source address, sends a response which can be hundreds of bytes in size to the machine it believes originated the request. The end result is that the victim’s network connection is hit with several hundred bytes of information that were not requested. They will be discarded when they reach the target machine, but not before exhausting a portion of the victim’s network bandwidth. And the traffic reaching the victim comes from the open resolver, not from the machine or machines used to initiate the attack. Given a large list of open resolvers to reflect against, an attacker using a DNS amplification attack can hide the origin of their attack and magnify the amount of traffic they can direct at the victim by a factor of 40 or more.
DNS operators who operate open resolvers without taking precautions to prevent their abuse generally believe they are harming nobody, but as the Spamhaus DDOS proves, open resolvers can be effortlessly coopted by attackers and used in criminal attacks on third parties.
Beginning in 2005, ISC began publicly advocating for operators to stop operating open resolvers. In 2007 we changed the behavior of BIND, the world’s most popular nameserver software, so that open resolvers would no longer be the default. And in 2008 ISC CTO Joao Damas co-authored RFC 5358, “Preventing Use of Recursive Nameservers in Reflector Attacks.” For 8 years now we’ve been consistently leading on this issue as part of our mission to strengthen the DNS infrastructure, improve network security, and contribute to a stable and open internet. But there are still many open resolvers in operation. In order to avoid being pressed into service as an unwitting pawn in a criminal conspiracy, ISC strongly advises that DNS operators make use of the security features in BIND to enforce reasonable access permissions on their recursive resolvers.
At ISC, supporting the health of the Domain Name System and improving the security and stability of an open internet are core values and the biggest part of our public mission. If you would like to know more about us and our efforts, follow us on Twitter or Linked In, or get in touch via our contact page.
Next week we’ll have more to share about how current ISC development efforts are targeting reflection attacks and other network abuse to create a better internet for everybody.
- By Adib Behjat on March 12, 2013
ORLANDO, FL–(Marketwire – Mar 12, 2013) – Internet Systems Consortium (ISC) is delighted to announce the launch of the Open Home Gateway Forum with an initial grant from Comcast. The OHGF is a Forum of ISPs and vendors and Internet development organizations, initiated and spearheaded by ISC, that aims to improve the rollout of new Internet technologies to home networks by providing stable, quality-assured reference Open Source software to be used in home gateways. Home gateways are the means by which a residential customer connects to an Internet service provider.
ISC has nearly two decades of experience in open source software development and the processes that surround it, including not just development, but release management, testing, bug management and support. ISC’s initial plans for home gateway software development include better support for IPv6, a smooth and secure mechanism to update the gateway’s firmware, and achieving better throughput and stability. Gateways are now first-class devices in the home, and they should have the software quality and functionality that people have come to expect from a first-class device.
“The edge device in homes is growing in function and importance,” said Kannan Ayyar, president of Internet Systems Consortium. “It must meet the needs of the consumer as well as the service provider. A home gateway needs to offer security, choice, speed, and flexibility. We are confident that the new devices enabled by the work of the Open Home Gateway Forum will be of tremendous value to Internet service customers as well as to their service providers.”
“Comcast is pleased that the Open Home Gateway Forum is one of the first programs underwritten with the new Comcast Technology Research & Development Fund,” said Jason Livingood, Vice President of Internet and Communications Engineering at Comcast. “We think this sort of open source development can make a meaningful difference in home networks and we’re committed to doing what we can to make the Forum a success.”
Further information about ISC’s Open Home Gateway project can be found at http://openhomegateway.org/
Internet Systems Consortium (ISC) is a 501(c)3 public benefit corporation widely known for world‐class Internet software engineering and network operations. Founded in 1994 under an initial grant from UUNET, ISC is governed today by a 5-member Board of Directors. ISC software, of which BIND and ISC DHCP are the two best‐known examples, is open source. Our passion is Internet core technology. Our widely‐imitated Managed Open Source process ensures the quality of our software while keeping it completely open and available. ISC operates high‐reliability global networks of DNS root servers (F‐root) and authoritative DNS servers both for non‐profit and commercial enterprises. ISC is actively involved in Internet protocol and standards development, particularly in the areas of DNSSEC and IPv6. ISC is supported by donations from generous sponsors, by program membership fees, and by increasing revenues from for-profit subsidiaries. For further information, please visit https://www.isc.org.
About Comcast Corporation
Comcast Corporation ( NASDAQ : CMCSA ) ( NASDAQ : CMCSK ) (www.comcast.com) is one of the nation’s leading providers of entertainment, information and communication products and services. With 23.8 million cable customers, 15.7 million high-speed Internet customers, and 7.4 million Comcast Digital Voice customers, Comcast is principally involved in the development, management and operation of cable systems and in the delivery of programming content.
Comcast’s content networks and investments include E! Entertainment Television, Style Network, Golf Channel, VERSUS, G4, PBS KIDS Sprout, TV One, ten sports networks operated by Comcast Sports Group and Comcast Interactive Media, which develops and operates Comcast’s Internet businesses, including Comcast.netContact:Media
+1 650 423 1327