request drops in BIND?

[Danny Mayer]

At 05:16 PM 5/12/2004, KyoungSoo Park wrote:

>>>>>Hi BIND hackers(or anyone interested in BIND's performance),
>>>>>
>>>>>To know BIND's reliability & performance, I tested BIND 9.2.3 by sending
>>>>>burst requests and
>>>>>seeing how many actually get back.
>>>>>When I did 200 requests of a single local domain name(the result should
>>>>>be cached in the BIND) at the same time,
>>>>>I see about 20 - 22 responses are usually missing.
>>>>>
>>>>>The BIND server that I configured is for testing only, so there is no
>>>>>user except for me.
>>>>>I wonder why it starts failing to respond at the burst request rate 178
>>>>>requests/sec.
>>>>>When I looked through tcpdump, then it seems the BIND gets all
>>>requests, but
>>>
>>>
>>>>>it seems it's dropping requests inside the server.
>>>>>
>>>>>
>>>>>
>>>>Are you sure about that?  It could be getting dropped in the UDP stack
>>>>*after* tcpdump has reported that the packet was received.  You should
>>>>enable BIND's debugging -- if it intentionally drops a request, I would
>>>>expect it to log a debugging message.
>>>>
>>>>
>>>>
>>>Yes. I'm quite positive. I did this many times, but see almost the same
>>>phenomenon.
>>>Getting dropped in the UDP stack doesn't seem to explain all, because
>>>all I did was to send 200 requests/sec at once
>>>(it's about 2 Mbps rate), and the total size of all 200 requests should
>>>be smaller than the recv buffer.
>>>Also, I did the same test using my UDP Ping, and it worked perfectly fine.
>>>
>>
>>Ping demands very little of a system. BIND needs to process the request
>>and send back a reply. They're hardly in the same league.
>>
>I said *UDP* ping that I made, not ICMP ping. My program receives the same 
>size of request(64byte) and
>sends back also the same size of response(about 330 bytes) as BIND to 
>simulate it.
>I don't see why BIND doesn't respond as fast as my UDP ping. What I expect 
>from BIND is
>to look up the result in the cache(less than 1 ms), then send the result 
>off. Why shouldn't BIND's
>response be as fast as my ping under such a small number of request burst?

It doesn't matter. your responses are making no demands on the system for 
looking up
and finding information. The only relationship is the roundtrip time. The 
time taken to
process the request is what you need to think about.

>>
>>
>>>Yes. I think it's a good idea to enable the debugging option and see
>>>what happens.
>>>
>>>
>>>
>>>>When we were having performance problems on our caching nameservers, we
>>>>noticed that queries to the virtual address that we publicized to our
>>>>customers would frequently time out, but queries sent to the server's
>>>>"real" address would have no problems.  So the problem was that the
>>>>per-socket buffer was overflowing (BIND uses a separate UDP socket for
>>>>each of the server's addresses).
>>>>
>>>>
>>>>
>>>What if there is only one IP address(no virtual address) for all the
>>>customers?
>>>64KB of default buffer seems not enough for handling bulk requests. It's
>>>not only used for getting requests
>>>but also used for getting responses from the remote name servers(such as
>>>gTDL servers). Those responses
>>>are usually big(300bytes), so it's possible that with only 218
>>>responses(=218*300 = 64KB), the buffer would fill up
>>>not even counting the requests' size. I think BIND's buffer should be at
>>>least a couple times bigger(ex. 10 times)
>>>to handle bulk requests. Bursty requests are always happening!
>>>
>>
>>That's 64KB PER REQUEST, not per socket. BIND processes messages on a
>>specific sockets and fills the buffer with one request at a time. Making 
>>the buffer
>>10 times larger makes absolutely no difference except to use up more memory.
>>What would help is having additional sockets, ie additional IP addresses 
>>to handle
>>the load.
>>
>What do you mean? The BIND source code explicitly checks the receive 
>buffer size at start,
>and sets it to 32KB when the default is not big enough. Most reasonable 
>unix's default buffer size
>is 64KB for a UDP socket. What do you mean by 64KB per request? How can 
>you make the receive buffer
>64KB for each incoming request? I think you might be confused with the 
>user level buffer to process a request
>and the kernel lever receive buffer per socket.

Your message didn't specify the difference. I assume you meant what's in 
the BIND code.
I don't remember what the default size is in socket.c, however, that buffer 
is per request.


>>BIND (on Unix) uses select() to handle multiple sockets and processes 
>>each socket
>>one at a time for all of the requests in the incoming socket and 
>>allocates the buffer for
>>receiving the data for each request. There is a mixture of tasks and 
>>events going on
>>inside BIND to handle all of this and make sure that request and reply 
>>buffers are set
>>up properly and that the request is processed.
>>
>The only client was me. My BIND program does nothing but responding to my 
>experiment requests.
>Also, the multiple sockets seem to be there to support virtual IP 
>addresses or tcp connections.

Yes, but if you are concerned about the buffers filling up before it has a 
chance to process
the request, it's easy to added IP addresses and thereby have more buffer 
available in
BIND.

>All in all, I don't see any good reasons why BIND should drop responses 
>whilemy UDP ping doesn't.
>I enabled the debug trace, but the trace looks so complex that I can't 
>clearly see what's going on.

Did you build multithreaded or singlethreaded? Is it a single CPU or multi-CPU?
How much memory does the machine have? How many domain records? Are the
queries for records for which the server is authorative or does it need to make
recursive queries if the answers are not in cache?

Yes the debug trace is complex and is unlikely to be easy for you to understand
especially if it's multithreaded. What is needed is to be able to look at 
the various
queues to see what they're doing.

Danny

>Thanks,
>KyoungSoo
>