How to log recursive queries?
Tim Peiffer
peiffer at umn.edu
Wed May 4 16:58:11 UTC 2005
Included inline is a perl script that I am using - I wrote it a while
back; it was inspired by a co-worker that
has long since left the University. You are free to use it however you
like. It will run on Bind8 or Bind9.
Note that I am not paying attention to the recursion desired bit... I
make decisions on what is in the scope
of the domains and networks I support. Anything outside of those bounds
is treated as external recursion.
Regards,
Tim Peiffer
Networking and Telecommunications Services
Univerisity of Minnesota
Configure your server into the arrays @OKNET and @OKDOM.
Add suitable logging lines in named.conf:
logging {
channel query_file {
file "/var/log/named/querylog" versions 15 size 200m; #
send to file
print-time yes;
severity info;
};
};
Add regular reporting using cron
30 06 * * * /usr/local/libexec/querystats -MailTo <my_report_destination>
40 06 * * * /usr/local/libexec/querystats -ExternalRecursive 1 -MailTo
<my_report_destination>
Example Report.
Report on MyServer named dated Wed Apr 27 06:45:01 2005
From files
/var/log/named/querylog.0
Report span 26-apr-2005 22:13:57 to 27-apr-2005 04:48:36 (23679 seconds)
Received and sent 2809734 transactions
Number of query clients 1304
Averaged 119 queries/second
Rank Client Addr query percent query
/Name count load /sec
1 24.163.240.83/CPE-24-163-240-83.mn.res.rr.com 52185 1.86 2.20
dev.null 3069
datacollect.com 2314
attbi.com 1687
mailbox.vhc.se 1680
thepalace.com 1679
mcsbsd.po.my 1679
katebww.com 1675
ibm.net 1667
vhc.se 1666
gracechurch-mn.org 1665
2 66.179.123.40/ns.arkware.net 11976 0.43 0.51
asandox.com 130
jaydemail.com 42
www.nytimes.com 29
view.atdmt.com 29
resalehost.networksolutions.com 28
img-cdn.mediaplex.com 18
a2.x.akamai.net 15
ad.3ad.doubleclick.net 15
a1470.g.akamai.net 15
spe.atdmt.com 15
3 66.179.123.43/ns4.arkware.net 9745 0.35 0.41
asandox.com 26
154.86.18.218.spam.dnsbl.sorbs.net 9
3.83.78.222.spam.dnsbl.sorbs.net 8
154.86.18.218.smtp.dnsbl.sorbs.net 7
142.217.33.59.spam.dnsbl.sorbs.net 7
119.10.173.61.spam.dnsbl.sorbs.net 6
201.22.172.61.smtp.dnsbl.sorbs.net 6
158.187.64.222.spam.dnsbl.sorbs.net 6
164.69.78.222.spam.dnsbl.sorbs.net 6
119.10.173.61.smtp.dnsbl.sorbs.net 5
#!/usr/bin/perl
#
# ACME Declarations Inc. This is necessary to satisfy strict();
#
my($result,$QueryDepth,$TopN, at OKNET, at OKDOM,%MONTH,$net,%PAT,$inaddr,$dom);
my($okpat, at query_files,$querylist,$listqueryfile,$client_IP,$linecount);
my($begintime,$endtime,$logline,$bind9,$linecount,$querysrc,$query);
my(%DNS_CLIENT,%DNS_QUERY_OBJECT,$what,$inst, at client_list,$NameAndIP);
my($MM,$DD,$hh,$mm,$ss,$YYYY,$mon,$year, at time, $total_queries,$querytotal);
my($i,$j,$host,$query_object,$file_handle,$fh,$External_Flag);
#
# Modules to use.
#
use Time::Local;
use Getopt::Long;
use strict;
use IO::Handle;
#use IO::File;
use IO::Pipe;
#
# Getopt variables to satisfy strict();
#
use vars qw/ $opt_TopN $opt_QueryDepth $opt_ExternalRecursive /;
use vars qw/ $opt_QueryFile $opt_man $opt_MailTo /;
#
# Extract various parameters being passed. These will affect
# depth and breadth of processing.
#
my($result) = GetOptions("TopN=i", "QueryDepth=i", "ExternalRecursive:i",
"QueryFile=s", "man:i", "MailTo=s");
#
# Extract command line arguments, or set reasonable defaults
#
if ($opt_QueryDepth) {
$QueryDepth = $opt_QueryDepth;
} else {
$QueryDepth = 10;
}
if ($opt_TopN) {
$TopN = $opt_TopN;
} else {
$TopN = 10;
}
if ($opt_man) {
$ENV{'SHELL'} = '/bin/sh';
$inst = '/usr/local/libexec';
if ($0 =~ /^[\.]*\//) {
exec "pod2man --section='l' $0 | groff -man -Tascii ";
} else {
exec "pod2man --section='l' $inst/$0 | groff -man -Tascii ";
}
exit 0;
}
if ($opt_MailTo) {
# $opt_MailTo =~ s/\@/\\\@/;
$fh = new IO::Pipe;
$result = $fh->writer("/usr/sbin/sendmail $opt_MailTo ");
} else {
$fh = new IO::Handle;
$result = $fh->fdopen(fileno(STDOUT), "w");
}
$file_handle = $result;
#
# This is a list of network and domain patterns. The network patterns
# will be used to construct suitable in-addr patterns.
#
@OKNET = (
# UMN
'128.101',
'131.212',
'134.84',
'146.57',
'160.94',
# RFC 1918
'192.168',
# GIGAPOP
'192.35.86',
'192.42.152',
# UWASH
'192.42.145',
'198.137.7',
'198.48.82',
'128.95',
'128.208',
'140.142',
'204.203',
'205.175'
);
@OKDOM = (
'umn.edu', 'gigapop.net', 'washington.edu'
);
#
# Set up hashed calendar for use with Time::Local
#
%MONTH = (
'jan', 0,
'feb', 1,
'mar', 2,
'apr', 3,
'may', 4,
'jun', 5,
'jul', 6,
'aug', 7,
'sep', 8,
'oct', 9,
'nov', 10,
'dec', 11
);
foreach $net (@OKNET) {
$PAT{$net} = 1;
$inaddr = join('.', reverse(split('.', $net)));
$inaddr .= "in-addr.arpa";
$PAT{$inaddr} = 1;
}
foreach $dom (@OKDOM) {
$PAT{$dom} = 1;
}
$okpat = join('|', keys %PAT);
@query_files = (
'/var/log/named/querylog.0',
);
if ($opt_QueryFile) {
@query_files = split(/,/, $opt_QueryFile);
$querylist = $opt_QueryFile;
} else {
$querylist = join(' ', @query_files);
}
$listqueryfile = join('\n\t', @query_files);
chomp($host = `hostname`);
if ($opt_ExternalRecursive) {
$External_Flag = "External Recursive";
}
#
# Now that things have been reasonably set up, we can spin through
# the query logs.
#
# The bind 8.x pattern to be parsed is:
# 04-Nov-2003 14:29:19.556 XX+/128.101.65.37/bento.software.umn.edu/A/IN
# The bind 9.x pattern to be parsed is:
# Nov 20 03:40:17.254 client 127.0.0.1#4220: query:
1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa
IN ANY
# Nov 20 03:40:45.972 client 10.0.0.1#2106: query: time.nist.gov IN A
#
$linecount = 0;
open(LOG, " cat $querylist |");
while(<LOG>) {
$logline = $_;
tr/A-Z/a-z/;
tr/\043/ /; # get rid of pesky '#' delimiter
$bind9 = 1 if (/query:/);
if ($linecount++ <= 1) {
if (($bind9 ==0) &&
(/^(.*)\.\d+ xx\+?/i)) {
$begintime = $1;
} elsif (/^(.*)\.\d+ client/i) {
$begintime = $1;
}
}
if (($bind9 ==0) && (/^(.*)\.\d+ xx\+?/i)) {
$endtime = $1;
} elsif (/^(.*)\.\d+ client/i) {
$endtime = $1;
}
if ($opt_ExternalRecursive) {
next if (/$okpat/i);
}
if ($bind9) {
($querysrc,$query) = (split(/\s+/))[4,7]
} else {
($querysrc,$query) = (split(/\//))[1,2];
}
$DNS_CLIENT{$querysrc} += 1;
$what = join('/', $querysrc, $query);
$DNS_QUERY_OBJECT{$what} += 1;
}
close(LOG);
$total_queries = $linecount;
#
# Render parse time into Unix time since the epoch
#
if ($bind9) {
($MM,$DD,$hh,$mm,$ss) =
($begintime =~ /(\S+) (\d+) (\d+):(\d+):(\d+)/);
$YYYY = 2003;
} else {
($DD,$MM,$YYYY,$hh,$mm,$ss) =
($begintime =~ /(\d+)-(\S+)-(\d+) (\d+):(\d+):(\d+)/);
}
$mon = $MONTH{$MM};
$year = $YYYY-1900;
$time[0] = timelocal($ss,$mm,$hh,$DD,$mon,$year);
if ($bind9) {
($MM,$DD,$hh,$mm,$ss) =
($endtime =~ /(\S+) (\d+) (\d+):(\d+):(\d+)/);
$YYYY = 2003;
} else {
($DD,$MM,$YYYY,$hh,$mm,$ss) =
($endtime =~ /(\d+)-(\S+)-(\d+) (\d+):(\d+):(\d+)/);
}
$mon = $MONTH{$MM};
$year = $YYYY-1900;
$time[1] = timelocal($ss,$mm,$hh,$DD,$mon,$year);
$time[2] = $time[1] - $time[0];
@client_list = keys %DNS_CLIENT;
#
# Send the report to STDOUT or a Mail destination if the MailTo option
# is set.
#
if ($opt_MailTo) {
printf $file_handle "From: dnsmon
To: $opt_MailTo
Subject: $host $External_Flag DNS Query log activity $begintime to $endtime
";
}
printf $file_handle "
Report on %s named dated %s
From files
%s
Report span %s to %s (%d seconds)
Received and sent %d transactions
Number of query clients %d
Averaged %4.0f queries/second
",
$host, scalar localtime(time), $listqueryfile,
$begintime, $endtime, $time[2], $total_queries, $#client_list+1,
$total_queries/$time[2];
printf $file_handle "%-5s %-50s %-5s %-5s %-5s\n",
"Rank", "Client Addr", "query", "percent", "query";
printf $file_handle "%-5s %-50s %-5s %-5s %-5s\n",
"", "/Name", "count", "load", "/sec";
foreach $client_IP (reverse sort srcnumeric %DNS_CLIENT) {
$i++;
last if ($DNS_CLIENT{$client_IP} == 0);
$NameAndIP = sprintf("%s/%s", $client_IP, &host($client_IP));
printf $file_handle "%5d %-50s %5d %5.2f %5.2f\n",
$i, $NameAndIP, $DNS_CLIENT{$client_IP},
$DNS_CLIENT{$client_IP}/$total_queries*100,
$DNS_CLIENT{$client_IP}/$time[2];
$j = 0;
foreach $query_object (reverse sort querynumeric grep(/$client_IP/,
keys %DNS_
QUERY_OBJECT)) {
$j++;
$querytotal = $DNS_QUERY_OBJECT{$query_object};
$query_object =~ s/.*\///g;
printf $file_handle "%5s %-50s %5d\n",
"", $query_object, $querytotal;
last if ($j >= $QueryDepth);
}
last if ($i >= $TopN);
}
#
# -1/0/1
# srcnumeric(a,b)
# querynumeric(a,b)
# Return -1/0/1 relationship between items a,b. These compare
# functions are used to drive sort();
#
sub srcnumeric {
$DNS_CLIENT{$a} <=> $DNS_CLIENT{$b};
}
sub querynumeric {
$DNS_QUERY_OBJECT{$a} <=> $DNS_QUERY_OBJECT{$b};
}
#
# string
# host(string ip address)
#
# Resolve the argument IP address into a DNS name.
#
sub host {
my($arg) = @_;
my($retn) = 'no_DNS_entry';
$ENV{'SHELL'} = '/bin/sh';
open(PIPE, "/usr/bin/host $arg 2>&1 |");
while(<PIPE>) {
if (/domain name pointer (\S+)/) {
$retn = $1;
last;
} elsif (/not found/) {
$retn = "no_DNS_entry";
last;
}
}
close(PIPE);
return $retn;
}
=head1 NAME
Querystats - Another DNS statistics package.
=head1 SYNOPSIS
querystats [-TopN <number>] [-QueryDepth <number>] [-ExternalRecursive]
-QueryFile=<file1>,<file2>...
=head1 DESCRIPTION
Querystats makes use of the bind query log to indicate where DNS traffic
is being generated, and what targets are most in use. This script would
be called daily to indicate TopN talkers, in order to get a handle on
whether the traffic is valid.
When passing a list of files to examine using -QueryFile, either quote the
list, or make sure there are no embedded spaces.
Traffic is not appropriate if it seems to be coming from a machine that
shows signs of compromise, including an open SMTP mail relay.
=head1 EXAMPLES
=item querystats -QueryFile /var/log/named/querylog.0
=item querystats -ExternalRecursive -QueryFile /var/log/named/querylog.0
=item querystats -TopN 20 -QueryFile
/var/log/named/querylog{.6,.5,.4,.3,.2,.1}
=back
=head1 FILES
F</var/log/named/querylog>
=head1 NOTES
There are two different formats used between Bind8.3 and Bind9.0. It
is expected that this script will automatically note which format is
in use.
The DNS services at the University of Minnesota produce appproximately
200MB each of querylog every 6 hours on the servers ns.nts and nss.nts.
With 1.4GHz Athalon servers running Linux, This script is able to digest
200MB in 5 minutes. Assuming Order(N) complexity, I believe that the
should complete in 1minute for every 40MB of query log.
=head1 DIAGNOSTICS
No diagnostics available.
=head1 BUGS
There are probably some bugs, but none known.
=head1 AUTHOR
Tim Peiffer, Networking and Telecom Services, University of Minnesota
=end
More information about the bind-users
mailing list