DNS query packets being generated by some rough process, can I trace the process?

[Kevin Darcy]

Alan Zoysa wrote:
> Hello.
> Am running a BIND-9.3.4-P1.1 on Debian stable.
>
> I find a continuous traffic from my machine to my ISP's DNS. Wireshark shows
> that the MAC address of the packets being sent have my network card's MAC
> address. So, I conclude that this traffic is being generated by some process
> on my machine.
>
> I want to know whether there is a way to figure out which
> process/application is generating a packet on Linux machine. A crude way can
> be to stop the running processes one-by-one and figure out which process was
> generating the traffic. But I do not intend to follow this method and want
> to know what triggered this scenario.
>
> It might be possible that some other machine on the subnet (served by a
> switch and not a hub) is generating these packets. If this is true and if
> the switch is sane I should not see the DNS query packets but only the
> response packets, right?
>
> I tried to use fuser command [ fuser 34423/udp ] to figure out if I can get
> the owner/process of the port number from which DNS queries are being send.
> The source port range being used is 34420-25.
>   
Any luck with that? I'm not familiar with "fuser" on Linux. On Solaris 
I'd use the open-source "lsof" command to map a particular socket port 
back to a particular process. I assume that the same kind of 
functionality has been rolled into Linux "fuser" program.

One approach is to "stall" these lookups long enough for "fuser" to 
catch them red-handed. Simply pointing /etc/resolv.conf to an arbitrary 
address probably wouldn't work very well for this, since the resolver 
might get a "connection refused", "no route to host" etc. and 
immediately proceed to the next nameserver in the list. It would be 
better to silently *discard* the first query, so that the stub resolver 
spends some time before proceeding to the next nameserver in the list. 
This can be accomplished by pointing the resolver to yourself and either 
a) mucking around with /etc/services, inetd etc. so that the "discard" 
(packet sink) service is listening on 53/udp, or b) running a dummy 
nameserver instance that discards the queries (with BIND you'd use the 
"blackhole" option for that).

Slowing down the lookups in such a way might buy you enough time to 
catch it. If that isn't enough, you might be able to twiddle with 
"options" in /etc/resolv.conf to increase the timeout even more. I'm 
assuming, perhaps incorrectly, that the resolver libraries on Linux 
understand /etc/resolv.conf "options". If not, you may be SOL in that 
regard.

Note that slowing down the lookups may cause them to actually *fail*. 
You can guarantee this, of course, by removing *all* valid nameserver 
addresses from /etc/resolv.conf. This might not be an option on a 24x7 
production system, but has the advantage, if the failure was 
sufficiently visible, of pinpointing the program/app which is generating 
all of this query traffic, even if "fuser" can't find it. It's like the 
old "yank the cable out and see who yells" method of network resource 
management :-)

You could, of course, approach this from the opposite end, performing a 
"pfiles" (or the Debian equivalent) on "likely suspect" processes, until 
you find the one that's responsible. The lookup-slowing or -failing 
technique(s) mentioned above might be useful for that as well.

                                                                         
                        - Kevin