DNS Capacity issue help -- Recursive Query -- it seems some packets are dropped by DNS

[Cathy Almond]

On 10/04/2018 01:37, PENG, JUNAN wrote:
> Hi, All
> 
> I did recursive query capacity test.   I used traffic generator to place 15K QPS traffic to DNS 1 with FQDN1 (Note, FQDN1 can't be resolve by DNS1, it need to forward it to DNS2  and TTL is set to 0)
> 
> But during the test , I found lots of failure , the successful rate is not high (85%).   Then I used TCPdump commands to capture logs in DNS1 , I found the following things:
> 
> Thing 1.  DNS query number is larger than response number between traffic generator and DNS1 .  About 15% traffic are dropped by DNS1 . 
> 
> Thing 2. DNS recursive query number between DNS1 and DNS2  is far less than query number between traffic generator and DNS1   

Tony Finch was correct earlier to point you in the direction of
max-clients-per-query.

There's also this KB article:

https://kb.isc.org/article/AA-00463/0/How-does-clients-per-query-work.html

But your test scenario is in any case flawed.  You're attempting to test
how well named can handle recursing every time, but that is not going to
happen because you're using the same FQDN.

What's happening here is that the first query received causes recursion
to commence to get the answer to the client query.  All the other
clients making the same query while this is ongoing, don't cause named
to start more recursion - instead they will be queued waiting for the
answer to be available (i.e. there are multiple clients per query at
this point in time).

When the answer comes back from recursion, it will be given to all those
clients that were waiting for it.  Then, because it had TTL=0, it's not
kept to be used for newer clients asking for the same thing -
essentially the process starts all over again.

And the other thing that is happening (as has already been pointed out)
is that you're (very likely) tripping up over the 'clients-per-query'
self-tuning throttle (designed to protect your server from a storm of
the same query from multiple clients).  This is going to result in
dropped queries.  Have a look at your logs (make sure you're logging
everything) - if you see clients-per-query being adjusted up and down,
then you've been hitting this limit.

Hope this info helps you to design a test that matches better to what
you need to achieve.

Cathy