Multiple Bell/Verizion DNS queries

[Brad Knowles]

At 11:49 AM -0400 4/17/01, Tom Nichols wrote:

>  This is coming through DNS. In this case, the Domain Name is registered
>  with our DNS servers. However, why should one query instigate a host of
>  queries from Bell Atlantic's DNS servers? So far today, there have been
>  1477 queries logged on this DNS server for this domain.

	Man, this is weird.  I've taken the lines you posted, stripped 
off the trailing bits that are all the same, and then re-folded them 
so that it's easier to see what's going on:

>  17-Apr-2001 01:13:49.386 queries: info: XX /141.151.128.68/
>  17-Apr-2001 01:13:49.419 queries: info: XX /141.151.128.68/
>  17-Apr-2001 01:13:49.452 queries: info: XX /141.151.128.68/
>  17-Apr-2001 01:13:49.559 queries: info: XX /141.151.128.68/

	These four queries appear to come from the same machine, and all 
within a span of about two-tenths of a second.

>  17-Apr-2001 01:14:21.883 queries: info: XX /199.45.32.51/
>  17-Apr-2001 01:14:21.909 queries: info: XX /199.45.32.51/
>  17-Apr-2001 01:14:21.933 queries: info: XX /199.45.32.51/
>  17-Apr-2001 01:14:22.037 queries: info: XX /199.45.32.51/

	Here's another group of four, and again within about two-tenths 
of a second.

>  17-Apr-2001 01:18:26.167 queries: info: XX /199.45.45.14/
>  17-Apr-2001 01:18:26.194 queries: info: XX /199.45.45.14/
>  17-Apr-2001 01:18:26.217 queries: info: XX /199.45.45.14/
>  17-Apr-2001 01:18:26.319 queries: info: XX /199.45.45.14/

	Yet another group of four, slightly more than two-tenths of a 
second separating the first from the last.

>  17-Apr-2001 01:19:29.365 queries: info: XX /141.154.0.68/
>  17-Apr-2001 01:19:29.394 queries: info: XX /141.154.0.68/
>  17-Apr-2001 01:19:29.421 queries: info: XX /141.154.0.68/
>  17-Apr-2001 01:19:29.558 queries: info: XX /141.154.0.68/

	Another group of four.

>  17-Apr-2001 01:22:05.198 queries: info: XX /151.197.0.68/
>  17-Apr-2001 01:22:05.221 queries: info: XX /151.197.0.68/
>  17-Apr-2001 01:22:05.242 queries: info: XX /151.197.0.68/
>  17-Apr-2001 01:22:05.319 queries: info: XX /151.197.0.68/

	And another group of four.

>  17-Apr-2001 01:22:06.374 queries: info: XX /141.155.0.68/
>  17-Apr-2001 01:22:06.410 queries: info: XX /141.155.0.68/
>  17-Apr-2001 01:22:06.510 queries: info: XX /141.155.0.68/

	A group of three?  Maybe one packet got dropped?

>  17-Apr-2001 01:23:03.558 queries: info: XX /141.150.0.68/
>  17-Apr-2001 01:23:03.589 queries: info: XX /141.150.0.68/
>  17-Apr-2001 01:23:03.697 queries: info: XX /141.150.0.68/

	Another group of three?

>  17-Apr-2001 01:23:49.476 queries: info: XX /141.151.128.68/
>  17-Apr-2001 01:23:49.511 queries: info: XX /141.151.128.68/
>  17-Apr-2001 01:23:49.541 queries: info: XX /141.151.128.68/
>  17-Apr-2001 01:23:49.661 queries: info: XX /141.151.128.68/

	Finally, another group of four.


	Looking up all these IP addresses, all but one appear to be from 
within the ba-dsg.net (Bell Atlantic Data Solutions Group) domain, 
and the other is also owned by Bell Atlantic.  Of course, you already 
knew this much.

	From looking at the names returned by the DNS, I'd guess that 
these are nameservers in cities like Pittsburgh, New York, Boston, 
etc....


	The one thing that strikes me as a possible reason why you could 
see four queries from the same IP address in such a short time is 
that they could be running a farm of four central caching nameservers 
at each site, and when you (or a customer) go visit a site at one of 
those locations, you might get your IP address looked up when they go 
to process their log data the next morning.

	If they have any intelligence, they will be running log 
processing software that internally caches IP address information, 
and they'll also have a local caching nameserver running on that 
machine, but they might forward all unknown queries to the farm of 
central caching nameservers, and if they don't handle the forwarding 
correctly (or if they don't have a local caching nameserver on the 
log-processing machine at all, and the application is written 
poorly), they may cause the same answer to be asked by multiple 
machines.

	Put this through a NAT, or run four copies of BIND 8, each bound 
to a separate processor of a machine with at least four CPUs, and you 
could see the source for all these queries being set to the same IP 
address (although they'd be using different port numbers).


	My guess is that someone at your site surfed a web site on their 
machines, but this web site was large enough to be geographically 
distributed to multiple locations around the world (in a manner 
similar to how Akamai works), and during the time of their session, 
they may have actually hit many different web servers.

	When it came time to process the logs from all these machines, 
they did so locally at each site (instead of centralizing the log 
files, and then effectively doing a "sort | uniq" before processing 
them), and with a poorly implemented nameservice infrastructure, they 
hit your nameserver far more times than you would otherwise expect.

	It could be even simpler -- if the log processing software 
doesn't have it's own internal caching but is multi-threaded (and 
they don't have a local caching nameserver running on the 
log-processing machine itself), the application could actually 
generate a very large number of unnecessary DNS queries, and that 
could be further amplified by poor DNS infrastructure on their side.



	You know, I'm thinking it would be really nice to have BIND log 
the source port as well as the source IP address in the case of log 
entries like this, so that it might be easier to tie something like 
this back to a situation like four copies of BIND running on the 
remote server, or alternatively going through some sort of NAT, 
etc....

	It would be even better if you could somehow get sequence numbers 
logged as well, but IIRC sequence numbers are not found in 
unreliable/connectionless protocols like UDP but only 
connection-oriented protocols like TCP.  ;-(

-- 
Brad Knowles, <brad.knowles at skynet.be>

/*        efdtt.c  Author:  Charles M. Hannum <root at ihack.net>          */
/*       Represented as 1045 digit prime number by Phil Carmody         */
/*     Prime as DNS cname chain by Roy Arends and Walter Belgers        */
/*                                                                      */
/*     Usage is:  cat title-key scrambled.vob | efdtt >clear.vob        */
/*   where title-key = "153 2 8 105 225" or other similar 5-byte key    */

dig decss.friet.org|perl -ne'if(/^x/){s/[x.]//g;print pack(H124,$_)}'