RPZ and www.rackspace.com

David A. Evans Evans_David_A at cat.com
Fri May 30 15:42:00 UTC 2014 

        To my question of how many more are lurking out there.  It looks 
like quite a few.  I am not sure we are going to be able to continue with 
RPZ's and NSDNAME's. 

         xserv.dell.com is my newest main stream web site having the 
issue.

        I is behaving the same way as www.rackspace.com.   They have DNS 
servers answering NXDOMAIN when they should be answering NODATA.






David A. Evans
Enterprise IP/DNS Management
Network Infrastructure Tools and Services
(309) 675-9700 
Evans_David_A at cat.com

bind-users-bounces at lists.isc.org wrote on 05/28/2014 03:09:54 PM:

> From: "David A. Evans" <Evans_David_A at cat.com>
> To: bind-users at isc.org
> Date: 05/28/2014 03:10 PM
> Subject: Re: RPZ and www.rackspace.com
> Sent by: bind-users-bounces at lists.isc.org
> 
> CATERPILLAR SECURITY ALERT: The email address in the sender line 
> does not match the account that sent the email. This can be an 
> indication of phishing. Do not click links or open attachments 
> unless you are certain it is from a safe source. Learn more at 
> security.cat.com/phishing
>         Rack Space appears to have fixed the issue.    "dig 
> www.wip.rackspace.com  NS"  now returns NO DATA instead of NXDOMAIN. 
> 
>         I wonder how many more are lurking out there. 
> 
> 
>         We are still getting a trickle in of complaints about 
> slowness and failures that appear to be related to the RPZ and the 
> amount of time it takes to complete all the extra queries for the 
> NSDNAME checks.   When we research these they seem to fit into 2 groups. 

> 
> 
>         1.   DNS zones with "slightly" broken infrastructure.  These
> would be domains with either slow response from one or more name 
> servers or not responding name servers.  A recursive resolver 
> without RPZ loaded can work though the issue but the extra lookups 
> required primarily for the NSDNAME increases the query time to the 
> point where DNS times out from the client perspective.    I can't 
> really see a fix here,  the issue does reside with the domain owner,
> we are simply more susceptible to the issue because of the RPZ's. 
> 
>         2.  DNS zones with a large number of NS records and the name
> servers have FQDN's in several different DNS zones. 
> 
> www.domain.com           NS        ns1.domain1.com 
>                         NS        ns2.domain2.com 
>                         NS        ns3.domain3.net  ..... 
> 
>         These are the most frustrating as there is really nothing 
> wrong with this setup in my opinion.   This is just going to 
> generate a huge number of DNS lookups to do a full NSDNAME check. 
> These are hard to explain away as they "work from home" and "work 
> from my phone". 
> 
> 
>         Have others found similar issues when implementing RPZ's? 
> 
> 
> 
> 
> 
> 
> David A. Evans 
> Enterprise IP/DNS Management 
> Network Infrastructure Tools and Services 
> 
> 
> 
> Mark Andrews <marka at isc.org> wrote on 05/07/2014 10:44:05 AM:
> 
> > From: Mark Andrews <marka at isc.org> 
> > To: "David A. Evans" <Evans_David_A at cat.com> 
> > Cc: bind-users at isc.org 
> > Date: 05/07/2014 10:44 AM 
> > Subject: Re: RPZ and www.rackspace.com 
> > 
> > 
> > In message <OFDC3C86D9.D668B707-ON86257CD1.005339FC-86257CD1.
> > 005431EB at notes.cat.com>, "David A. Evans" writes:
> > > 
> > >         I've done some more troubleshooting with info from people 
that 
> > > responded directly to me and not to the list.    This can be 
reproduced 
> > > without any RPZ loaded by mimicking the behavior of the RPZ lookups 
> > > required to validate NSDNAME lines.
> > > 
> > > Issue these 'digs' within 30 second of each other.
> > > 
> > > dig www.wip.rackspace.com 
> > > www.wip.rackspace.com.  30      IN      A       173.203.44.116 
> > > 
> > > dig www.wip.rackspace.com  NS
> > > (NXDOMAIN)
> > > 
> > > dig www.wip.rackspace.com
> > > (NXDOMAIN)
> > > 
> > > 
> > >         I think this is another case of miss configured load 
balancers. 
> > > Shouldn't the NS record lookup respond with a NODATA response and 
not 
> > > NXDOMAIN? 
> > 
> > Yes.  The name exists.
> > 
> > >         That still doesn't really answer why a site as big as 
> > > www.rackspace.com isn't getting hit with support issues on theirweb 
site. 
> > >  It only took us about 4 hours into our first production day with 
> > > NSDNAME's in our RPZ to get a call about www.rackspace.com not 
loading.
> > 
> > Because NS queries are not common with normal DNS lookups.  For
> > some reason people that deploy load balancers think they don't need
> > to fix issues like this.  Send something other than a A record and
> > you get:
> > 
> >    - NXDOMAIN being returned when the name exists.
> >    - NOTIMP being returned.
> >      (Really you can't just send NODATA?)
> >    - REFUSED being returned.
> >      (Really you don't want to tell us the record does not exist?)
> >    - the wrong SOA being returned.
> >    - malformed RDATA with the content being the A record content.
> > 
> > Mark
> > 
> > > David A. Evans
> > > Enterprise IP/DNS Management
> > > Network Infrastructure Tools and Services
> > > Evans_David_A at cat.com
> > > 
> > > 
> > > 
> > > From:   "David A. Evans" <Evans_David_A at cat.com>
> > > To:     bind-users at lists.isc.org
> > > Date:   05/07/2014 09:11 AM
> > > Subject:        RPZ and www.rackspace.com
> > > Sent by:        bind-users-bounces at lists.isc.org
> > > 
> > > 
> > > 
> > > CATERPILLAR SECURITY ALERT: The email address in the sender linedoes 
not 
> > > match the account that sent the email. This can be an indication of 
> > > phishing. Do not click links or open attachments unless you are 
> certain it 
> > > is from a safe source. Learn more at security.cat.com/phishing
> > >         We have just enabled RPZ with some NSDNAME checks and are 
seeing 
> > > an issue resolving www.rackspace.com. 
> > > 
> > >         The first lookup is successful and returns both the CNAME 
and the 
> > > A record.  The second query, within a second of the first, will only 

> > > return the CNAME.  It will only return the CNAME until the TTL of 
the A 
> > > record times out.  The first query, when it actually has to go out 
and do 
> > > recursion will always work.   Answering from cache will always fail. 
When 
> > > you inspect the cache during the time that it is only returning 
> the CNAME, 
> > > the record in cache is "www.wip.rackspace.com  type ANY 
> NXDOMAIN".    This 
> > > only happens with RPZ's enabled and query hitting a RPZ zone with a 
> > > NSDNAME line.   Turning off RPZ or whitelisting the lookup via RPZ 
before 
> > > it hits a RPZ with NSDNAME allows the query to be successful 100% of 
the 
> > > time. 
> > > 
> > > 
> > >         Can anyone else verify this behavior?   What is going on 
with 
> > > www.rackspace.com?   If this is a miss configuration on Rackspace's 
DNS 
> > > servers how are they not getting hit with support calls like crazy? 
> > > 
> > > 
> > > 
> > > dig @redacted.cat.com www.rackspace.com 
> > > 
> > > ; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6_4.6 <<>> 
@redacted.cat.com 
> > > www.rackspace.com 
> > > ; (1 server found) 
> > > ;; global options: +cmd 
> > > ;; Got answer: 
> > > ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30337 
> > > ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0 

> > > 
> > > ;; QUESTION SECTION: 
> > > ;www.rackspace.com.             IN      A 
> > > 
> > > ;; ANSWER SECTION: 
> > > www.rackspace.com.      300     IN      CNAME   
www.wip.rackspace.com. 
> > > www.wip.rackspace.com.  30      IN      A       173.203.44.116 
> > > 
> > > ;; Query time: 193 msec 
> > > ;; SERVER: redacted 
> > > ;; WHEN: Wed May  7 08:53:08 2014 
> > > ;; MSG SIZE  rcvd: 73 
> > > 
> > > 
> > > 
> > > dig @redacted.cat.com www.rackspace.com 
> > > 
> > > ; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6_4.6 <<>> 
@redacted.cat.com 
> > > www.rackspace.com 
> > > ; (1 server found) 
> > > ;; global options: +cmd 
> > > ;; Got answer: 
> > > ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 25905 
> > > ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 0 

> > > 
> > > ;; QUESTION SECTION: 
> > > ;www.rackspace.com.             IN      A 
> > > 
> > > ;; ANSWER SECTION: 
> > > www.rackspace.com.      298     IN      CNAME   
www.wip.rackspace.com. 
> > > 
> > > ;; AUTHORITY SECTION: 
> > > wip.rackspace.com.      58      IN      SOA 
www-gtm-ord1.rackspace.com. 
> > > hostmaster.305181-GTM1.rackspace.com. 86 10800 3600 604800 60 
> > > 
> > > ;; Query time: 2 msec 
> > > ;; SERVER: redacted 
> > > ;; WHEN: Wed May  7 08:53:10 2014 
> > > ;; MSG SIZE  rcvd: 129 
> > > 
> > > 
> > > David A. Evans 
> > > Enterprise IP/DNS Management 
> > > Network Infrastructure Tools and Services 
> > > Evans_David_A at cat.com 
_______________________________________________
> > > Please visit https://lists.isc.org/mailman/listinfo/bind-users to 
> > > unsubscribe from this list
> > > 
> > > bind-users mailing list
> > > bind-users at lists.isc.org
> > > https://lists.isc.org/mailman/listinfo/bind-users
> > > 
> > > --=_alternative 0054318186257CD1_=
> > > Content-Type: text/html; charset="US-ASCII"
> > > _______________________________________________
> Please visit https://lists.isc.org/mailman/listinfo/bind-users to 
> unsubscribe from this list
> 
> bind-users mailing list
> bind-users at lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/bind-users/attachments/20140530/f36a4219/attachment.html>

More information about the bind-users mailing list