Too many open files Re: Patch 9.4.2->9.4.2-P1 breaking tcp-responder?

[jbratton at rackspace.com]

Quoting Thomas Jacob <jacob at internet24.de>:

> That didn't seem to do it for us, the bind instance in question ran
> for about 38 hours and then it refused to accept tcp connections again.

Did it actually start building up a lot of TCP connections in SYN_RECV  
state again, or did it just crash?

The reason I ask is because my busiest caches will crash after running  
for a day or two with the following entries in the logs:

Jul 27 13:15:07 s_all at cachens-3/69.20.2.254 named[24864]:  
socket.c:1736: INSIST(!sock->pending_recv) failed
Jul 27 13:15:07 s_all at cachens-3/69.20.2.254 named[24864]: exiting (due  
to assertion failure)

If anybody would benefit from seeing the gdb output, let me know.  I  
have *plenty* of core files laying around now.

> I found the following error message in the logs at about the
> time of the outage:
>
> 27-Jul-2008 15:35:12.234 resolver: notice: clients-per-query decreased to 17
> 27-Jul-2008 15:35:34.440 general: error: socket.c:1996: unexpected error:
> 27-Jul-2008 15:35:34.440 general: error: internal_accept: fcntl()   
> failed: Too many open files
> 27-Jul-2008 15:35:34.452 general: error: socket.c:1996: unexpected error:
> 27-Jul-2008 15:35:34.452 general: error: internal_accept: fcntl()   
> failed: Too many open files

You can fix that with ulimit.  Check out ulimit -n to see how many  
open files you currently allow, and try increasing it.  I keep it set  
to 16384 on my busier caches without any issues.  Note that setting  
the limit with ulimit won't be persistent, you will want to change  
/etc/security/limits.conf and add something like the following:

*               soft    nofile          16384
*               hard    nofile          16384

Of course, this changes the limits for everything, but since mine are  
dedicated nameservers with almost nothing else running on them, this  
isn't an issue.

Hope this helps.

-- Jason


Confidentiality Notice: This e-mail message (including any attached or
embedded documents) is intended for the exclusive and confidential use of the
individual or entity to which this message is addressed, and unless otherwise
expressly indicated, is confidential and privileged information of Rackspace.
Any dissemination, distribution or copying of the enclosed material is prohibited.
If you receive this transmission in error, please notify us immediately by e-mail
at abuse at rackspace.com, and delete the original message.
Your cooperation is appreciated.