RPZ and www.rackspace.com
David A. Evans
Evans_David_A at cat.com
Wed May 28 20:09:54 UTC 2014
Rack Space appears to have fixed the issue. "dig
www.wip.rackspace.com NS" now returns NO DATA instead of NXDOMAIN.
I wonder how many more are lurking out there.
We are still getting a trickle in of complaints about slowness and
failures that appear to be related to the RPZ and the amount of time it
takes to complete all the extra queries for the NSDNAME checks. When we
research these they seem to fit into 2 groups.
1. DNS zones with "slightly" broken infrastructure. These would
be domains with either slow response from one or more name servers or not
responding name servers. A recursive resolver without RPZ loaded can work
though the issue but the extra lookups required primarily for the NSDNAME
increases the query time to the point where DNS times out from the client
perspective. I can't really see a fix here, the issue does reside with
the domain owner, we are simply more susceptible to the issue because of
the RPZ's.
2. DNS zones with a large number of NS records and the name
servers have FQDN's in several different DNS zones.
www.domain.com NS ns1.domain1.com
NS ns2.domain2.com
NS ns3.domain3.net .....
These are the most frustrating as there is really nothing wrong
with this setup in my opinion. This is just going to generate a huge
number of DNS lookups to do a full NSDNAME check. These are hard to
explain away as they "work from home" and "work from my phone".
Have others found similar issues when implementing RPZ's?
David A. Evans
Enterprise IP/DNS Management
Network Infrastructure Tools and Services
Mark Andrews <marka at isc.org> wrote on 05/07/2014 10:44:05 AM:
> From: Mark Andrews <marka at isc.org>
> To: "David A. Evans" <Evans_David_A at cat.com>
> Cc: bind-users at isc.org
> Date: 05/07/2014 10:44 AM
> Subject: Re: RPZ and www.rackspace.com
>
>
> In message <OFDC3C86D9.D668B707-ON86257CD1.005339FC-86257CD1.
> 005431EB at notes.cat.com>, "David A. Evans" writes:
> >
> > I've done some more troubleshooting with info from people that
> > responded directly to me and not to the list. This can be
reproduced
> > without any RPZ loaded by mimicking the behavior of the RPZ lookups
> > required to validate NSDNAME lines.
> >
> > Issue these 'digs' within 30 second of each other.
> >
> > dig www.wip.rackspace.com
> > www.wip.rackspace.com. 30 IN A 173.203.44.116
> >
> > dig www.wip.rackspace.com NS
> > (NXDOMAIN)
> >
> > dig www.wip.rackspace.com
> > (NXDOMAIN)
> >
> >
> > I think this is another case of miss configured load
balancers.
> > Shouldn't the NS record lookup respond with a NODATA response and not
> > NXDOMAIN?
>
> Yes. The name exists.
>
> > That still doesn't really answer why a site as big as
> > www.rackspace.com isn't getting hit with support issues on their web
site.
> > It only took us about 4 hours into our first production day with
> > NSDNAME's in our RPZ to get a call about www.rackspace.com not
loading.
>
> Because NS queries are not common with normal DNS lookups. For
> some reason people that deploy load balancers think they don't need
> to fix issues like this. Send something other than a A record and
> you get:
>
> - NXDOMAIN being returned when the name exists.
> - NOTIMP being returned.
> (Really you can't just send NODATA?)
> - REFUSED being returned.
> (Really you don't want to tell us the record does not exist?)
> - the wrong SOA being returned.
> - malformed RDATA with the content being the A record content.
>
> Mark
>
> > David A. Evans
> > Enterprise IP/DNS Management
> > Network Infrastructure Tools and Services
> > Evans_David_A at cat.com
> >
> >
> >
> > From: "David A. Evans" <Evans_David_A at cat.com>
> > To: bind-users at lists.isc.org
> > Date: 05/07/2014 09:11 AM
> > Subject: RPZ and www.rackspace.com
> > Sent by: bind-users-bounces at lists.isc.org
> >
> >
> >
> > CATERPILLAR SECURITY ALERT: The email address in the sender line does
not
> > match the account that sent the email. This can be an indication of
> > phishing. Do not click links or open attachments unless you are
certain it
> > is from a safe source. Learn more at security.cat.com/phishing
> > We have just enabled RPZ with some NSDNAME checks and are
seeing
> > an issue resolving www.rackspace.com.
> >
> > The first lookup is successful and returns both the CNAME and
the
> > A record. The second query, within a second of the first, will only
> > return the CNAME. It will only return the CNAME until the TTL of the
A
> > record times out. The first query, when it actually has to go out and
do
> > recursion will always work. Answering from cache will always fail.
When
> > you inspect the cache during the time that it is only returning the
CNAME,
> > the record in cache is "www.wip.rackspace.com type ANY NXDOMAIN".
This
> > only happens with RPZ's enabled and query hitting a RPZ zone with a
> > NSDNAME line. Turning off RPZ or whitelisting the lookup via RPZ
before
> > it hits a RPZ with NSDNAME allows the query to be successful 100% of
the
> > time.
> >
> >
> > Can anyone else verify this behavior? What is going on with
> > www.rackspace.com? If this is a miss configuration on Rackspace's
DNS
> > servers how are they not getting hit with support calls like crazy?
> >
> >
> >
> > dig @redacted.cat.com www.rackspace.com
> >
> > ; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6_4.6 <<>>
@redacted.cat.com
> > www.rackspace.com
> > ; (1 server found)
> > ;; global options: +cmd
> > ;; Got answer:
> > ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30337
> > ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
> >
> > ;; QUESTION SECTION:
> > ;www.rackspace.com. IN A
> >
> > ;; ANSWER SECTION:
> > www.rackspace.com. 300 IN CNAME www.wip.rackspace.com.
> > www.wip.rackspace.com. 30 IN A 173.203.44.116
> >
> > ;; Query time: 193 msec
> > ;; SERVER: redacted
> > ;; WHEN: Wed May 7 08:53:08 2014
> > ;; MSG SIZE rcvd: 73
> >
> >
> >
> > dig @redacted.cat.com www.rackspace.com
> >
> > ; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6_4.6 <<>>
@redacted.cat.com
> > www.rackspace.com
> > ; (1 server found)
> > ;; global options: +cmd
> > ;; Got answer:
> > ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 25905
> > ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 0
> >
> > ;; QUESTION SECTION:
> > ;www.rackspace.com. IN A
> >
> > ;; ANSWER SECTION:
> > www.rackspace.com. 298 IN CNAME www.wip.rackspace.com.
> >
> > ;; AUTHORITY SECTION:
> > wip.rackspace.com. 58 IN SOA
www-gtm-ord1.rackspace.com.
> > hostmaster.305181-GTM1.rackspace.com. 86 10800 3600 604800 60
> >
> > ;; Query time: 2 msec
> > ;; SERVER: redacted
> > ;; WHEN: Wed May 7 08:53:10 2014
> > ;; MSG SIZE rcvd: 129
> >
> >
> > David A. Evans
> > Enterprise IP/DNS Management
> > Network Infrastructure Tools and Services
> > Evans_David_A at cat.com _______________________________________________
> > Please visit https://lists.isc.org/mailman/listinfo/bind-users to
> > unsubscribe from this list
> >
> > bind-users mailing list
> > bind-users at lists.isc.org
> > https://lists.isc.org/mailman/listinfo/bind-users
> >
> > --=_alternative 0054318186257CD1_=
> > Content-Type: text/html; charset="US-ASCII"
> >
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/bind-users/attachments/20140528/01d0fbdd/attachment-0001.html>
More information about the bind-users
mailing list