DNS Capacity issue help -- Recursive Query -- it seems some packets are dropped by DNS

[PENG, JUNAN]

Hi, Tony and Cathy

Yes,  you are right.   It is caused by query using same FQDN and TTL=0.  I went to adjust  'clients-per-query' and ' max-clients-per-query' parameters during the test, there was a big difference. 

I also saw clients-per-query being adjusted up and down in logs :)   anyway,  I am looking for multiple FQDNs solution to alleviate it.

Thank you very much!

BR


-----Original Message-----
From: bind-users [mailto:bind-users-bounces at lists.isc.org] On Behalf Of Cathy Almond
Sent: Friday, April 13, 2018 4:14 AM
To: bind-users at lists.isc.org
Subject: Re: DNS Capacity issue help -- Recursive Query -- it seems some packets are dropped by DNS

On 10/04/2018 01:37, PENG, JUNAN wrote:
> Hi, All
> 
> I did recursive query capacity test.   I used traffic generator to place 15K QPS traffic to DNS 1 with FQDN1 (Note, FQDN1 can't be resolve by DNS1, it need to forward it to DNS2  and TTL is set to 0)
> 
> But during the test , I found lots of failure , the successful rate is not high (85%).   Then I used TCPdump commands to capture logs in DNS1 , I found the following things:
> 
> Thing 1.  DNS query number is larger than response number between traffic generator and DNS1 .  About 15% traffic are dropped by DNS1 . 
> 
> Thing 2. DNS recursive query number between DNS1 and DNS2  is far less than query number between traffic generator and DNS1   

Tony Finch was correct earlier to point you in the direction of max-clients-per-query.

There's also this KB article:

https://urldefense.proofpoint.com/v2/url?u=https-3A__kb.isc.org_article_AA-2D00463_0_How-2Ddoes-2Dclients-2Dper-2Dquery-2Dwork.html&d=DwICAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=xVh2hygmrxdOcVtuPuDNjQ&m=L93rGQDGg9_j4oNxcd_ghnG5KRYTElry1B5GJf6e_PU&s=yhxZMriJCxLZZJZ7o5ulNZbdtvwXlcJDhavl-FhCApA&e= 

But your test scenario is in any case flawed.  You're attempting to test how well named can handle recursing every time, but that is not going to happen because you're using the same FQDN.

What's happening here is that the first query received causes recursion to commence to get the answer to the client query.  All the other clients making the same query while this is ongoing, don't cause named to start more recursion - instead they will be queued waiting for the answer to be available (i.e. there are multiple clients per query at this point in time).

When the answer comes back from recursion, it will be given to all those clients that were waiting for it.  Then, because it had TTL=0, it's not kept to be used for newer clients asking for the same thing - essentially the process starts all over again.

And the other thing that is happening (as has already been pointed out) is that you're (very likely) tripping up over the 'clients-per-query'
self-tuning throttle (designed to protect your server from a storm of the same query from multiple clients).  This is going to result in dropped queries.  Have a look at your logs (make sure you're logging
everything) - if you see clients-per-query being adjusted up and down, then you've been hitting this limit.

Hope this info helps you to design a test that matches better to what you need to achieve.

Cathy

_______________________________________________
Please visit https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.isc.org_mailman_listinfo_bind-2Dusers&d=DwICAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=xVh2hygmrxdOcVtuPuDNjQ&m=L93rGQDGg9_j4oNxcd_ghnG5KRYTElry1B5GJf6e_PU&s=kddAM2ISqC6rwxchE8ZvKOfPHq_mT5vuLqUDBHBJD40&e=  to unsubscribe from this list

bind-users mailing list
bind-users at lists.isc.org
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.isc.org_mailman_listinfo_bind-2Dusers&d=DwICAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=xVh2hygmrxdOcVtuPuDNjQ&m=L93rGQDGg9_j4oNxcd_ghnG5KRYTElry1B5GJf6e_PU&s=kddAM2ISqC6rwxchE8ZvKOfPHq_mT5vuLqUDBHBJD40&e=