DNS server caching performance test results.

[Brad Knowles]

At 6:12 PM -0600 5/16/01, Matt Simerson wrote:

>  Configuring the caches was pretty basic. Run unlimit for BIND 8 and BIND 9
>  and add a couple lines to the options section (forward only; forwarders {
>  216.122.x.x; };) to convince it to only query the dns wall. Dnscache was
>  also easy to configure: (echo "216.122.x.x" >
>  ~dnscache/root/servers/122.216.in-addr.arpa.). That tells dnscache to
>  forward all requests for the 216.122 network off to dnswall.

	IMO, forwarding is almost always a problem.  I think a better 
trick would have been to configure the caching nameservers to be 
authoritative for the 216.in-addr.arpa zone, and delegate 
122.216.in-addr.arpa to the "walldns" machine, at which point the 
data would be cached.  So long as you restrict yourself to just PTR 
queries within this zone, I believe that this would be a cleaner 
test, and would avoid potentially problematical forwarding code or 
accidental improper configuration of forwarding.

>  Dnsfilter is a C program supplied with djbdns that takes a list of IP's and
>  does lookups on them. It writes the output back to STDOUT and I piped all
>  the output to files to verify the accuracy of the results. After much
>  testing of dnsfilter and it's limitations, I deduced that setting it's
>  number of parallel lookups higher than 100 effectively chokes it after
>  around 12,000 very quick queries. Keeping the number low prevented that. I
>  ran most tests at the default value of 10 paralled lookups unless otherwise
>  noted. The only reason to use a -c value higher than 100 is when querying
>  real world data where you need lots more parallel lookups because you'll
>  have a high number of time-outs or other failures.

	I'd be real interested to see a graph of number of parallel 
threads against total time to resolve the queries.  You might be 
surprised to find that the optimal number of threads was five, or it 
might turn out that the optimal number of threads is twenty-five. 
But only by testing the whole range of possibilities in the various 
test cases below, would you be likely to find out.

>  What follows is the output of my first batch of tests. I ran the following
>  command 3 times for each dns cache: "time dnsfilter < iplist.wall >
>  out[1-3]". The first test reflects the caches need to fetch the results from
>  the dns wall and return them to the client. The two subsequent tests reflect
>  the caches ability to server results from it's cache. The file iplist.wall
>  simply contains 65,536 ip addresses representing the class B address of
>  216.122.0.0.

	This is with just one client, right?

>  Name Server               time(s)   qps
>  dnscache - 290MB RAM      63        1040
>                            29        2260
>                            29        2260
>
>  BIND 8 - 8MB RAM          21        3120
>                            40        1638
>                            39        1680
>
>  BIND 9 - 12MB RAM         81        809
>                            29        2260
>                            29        2260

	Fascinating.  I wonder if BIND 8 is returning the answer before 
storing it in the cache, while the other two programs are storing it 
in the cache first and then returning the answer?

	Have you looked at "dents" as another alternative nameserver program?

>  Memory usage isn't meaningful for dnscache as it's a startup parameter. You
>  tell it how big a cache you want to maintain and once it's full it throws
>  out the oldest entries. I consider that to be better than allowing your
>  cache to grow until it exhausts all your physical RAM and swap (which I do
>  later :-)).

	I agree that it is better to specify the maximum, but IMO it 
would be less confusing to display the starting memory utilized, the 
ending memory utilized, and then in a footnote indicate what memory 
limitation parameters there may have been supplied.  I believe that 
the focus should be on the actual utilization, and not on the 
parameters.

>  The next thing I did was to spread out the client loads. I did this by
>  splitting the file "iplist.wall" into three equal sized chunks and copying
>  them to three servers with dnsfilter installed (hardware specs the same for
>  all dnsclient servers). So, each dns client would be responsible for looking
>  up about twenty thousand IP addresses. I then executed the following command
>  on all three servers at the same time: "time dnsfilter <
>  /usr/iplist.wall.seg > /usr/out[1-3]". Client time is the combined time
>  spent by all three clients looking up data. Time is elapsed time taken to
>  run the test (in seconds). Here are the results:

	To get a better idea of the steady-state performance of these 
programs, I'd like to see two clients continually running through the 
entire list of IP addresses, as quickly as they possibly can (no 
timing, direct all output to /dev/null, etc...).  Then, once they've 
cycled through at least a couple of times, fire up a third client 
which runs through the entire list of IP addresses several times, but 
this time you keep track of the output and the times.  You would then 
throw out the highest and lowest numbers, and average the rest.  I'd 
encourage at least seven runs like this, and the more the better.

>  Name Server                client-time    time      qps
>      dnscache - 290MB RAM   200             67       981
>                             93              31       2112
>                             86              29       2286
>
>      BIND 8 - 8MB RAM       51              17       3855
>                             114             38       1725
>                             114             38       1725
>
>      BIND 9 - 12MB RAM      239             80       822
>                             82              27       2397
>                             81              27       2427

	Even so, this is a very interesting set of results.  Given what 
Rick Jones has found out about bottlenecking in malloc() in BIND 9 
under stress, I have to believe that work is being done in this area 
for upcoming releases of BIND 9, and we should see these numbers 
increase rather dramatically over what you've got here.

>  I've ran the tests a couple more times and got similar results. I'm fairly
>  confident that I've reached the maximum abilities of each dns server on the
>  current hardware. I'm also quite confident that the testing is yielding
>  accurate results.

	Previously, you gave us information about system utilization on 
both client and server during the tests -- is this information 
available for this run?  I'd be very interested to know how hard the 
various programs ended up pushing the system in order to deliver the 
numbers you've gotten.


	Also, have you considered running this kind of test where the 
nameserver was running on a multi-processor machine (ideally both 
dual-processor and quad-processor)?  I'd be really interested to see 
how these numbers change when you throw extra processors into the mix.


	Thanks for this very interesting information!  I look forward to 
seeing further results of your testing!

-- 
Brad Knowles, <brad.knowles at skynet.be>

/*        efdtt.c  Author:  Charles M. Hannum <root at ihack.net>          */
/*       Represented as 1045 digit prime number by Phil Carmody         */
/*     Prime as DNS cname chain by Roy Arends and Walter Belgers        */
/*                                                                      */
/*     Usage is:  cat title-key scrambled.vob | efdtt >clear.vob        */
/*   where title-key = "153 2 8 105 225" or other similar 5-byte key    */

dig decss.friet.org|perl -ne'if(/^x/){s/[x.]//g;print pack(H124,$_)}'