random dispatch error after reconfig with bind ESV 9.11
Laurent Frigault
lolo at troll.free.org
Wed Jun 24 11:11:25 UTC 2020
Hi,
OS: FreeBSD 12.1 / AMD64
56 cores CPU
64G RAM
bind ESV 9.11.19 from standard freebsd pkg
We are runng an hiddenmaster bind server with DNSSEC zone "bookmyname.be" {
zone "bookmyname.be" {
type master;
file "custom/b/o/bookmyname.be/bookmyname.be";
notify explicit;
also-notify { 213.36.252.135; 62.210.98.15; 213.36.253.14; };
auto-dnssec maintain;
inline-signing yes;
key-directory "custom/b/o/bookmyname.be";
};
We have about 73000 zones , most signed.
We periodically regenerate our configuration to add/update/remove zones.
when needed, we use "rndc reconfig"
Every few weeks we get the following error in the log :
Jun 18 11:02:41 nsmaster named[50196]: 18-Jun-2020 11:02:41.989 general: error: ./server.c:3881: unexpected error:
Jun 18 11:02:41 nsmaster named[50196]: 18-Jun-2020 11:02:41.989 general: error: unable to obtain neither an IPv4 nor an IPv6 dispatch
Jun 18 11:02:42 nsmaster named[50196]: 18-Jun-2020 11:02:42.728 general: error: reloading configuration failed: unexpected error
And after this, it stops responding for some zone but not all of them
and not always the recently added.
Of course the ipv4 and v6 where not changed/added/remove on the server and th configuration was correct it is generated by a script from a database).
After this, the only solution is to stop and restart bind. An other rndc reconfig produce the same error.
Someone tells me this may be a socket issue.
The freebsd pkg/port is build with --with-tuning=default
% /usr/local/sbin/named -V
BIND 9.11.19 (Extended Support Version) <id:905ec64>
running on FreeBSD amd64 12.1-RELEASE-p6 FreeBSD 12.1-RELEASE-p6 GENERIC
built by make with '--localstatedir=/var' '--disable-linux-caps' '--with-randomdev=/dev/random' '--with-libxml2=/usr/local' '--with-readline=-L/usr/local/lib -ledit' '--with-dlopen=yes' '--with-gost=no' '--without-python' '--sysconfdir=/usr/local/etc/namedb' '--with-dlz-filesystem=yes' '--disable-dnstap' '--enable-filter-aaaa' '--disable-fixed-rrset' '--without-geoip2' '--without-gssapi' '--with-libidn2=/usr/local' '--enable-ipv6' '--with-libjson=/usr/local' '--disable-largefile' '--with-lmdb=/usr/local' '--disable-native-pkcs11' '--disable-querytrace' '--enable-rpz-nsdname' '--enable-rpz-nsip' 'STD_CDEFINES=-DDIG_SIGCHASE=1' '--with-openssl=/usr' '--enable-threads' '--with-tuning=default' '--disable-symtable' '--prefix=/usr/local' '--mandir=/usr/local/man' '--infodir=/usr/local/share/info/' '--build=amd64-portbld-freebsd12.1' 'build_alias=amd64-portbld-freebsd12.1' 'CC=cc' 'CFLAGS=-O2 -pipe -DLIBICONV_PLUG -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing ' 'LDFLAGS= -fstack-protector-strong ' 'LIBS=-L/usr/local/lib' 'CPPFLAGS=-DLIBICONV_PLUG -isystem /usr/local/include' 'CPP=cpp' 'PKG_CONFIG=pkgconf'
compiled by CLANG 4.2.1 Compatible FreeBSD Clang 8.0.1 (tags/RELEASE_801/final 366581)
compiled with OpenSSL version: OpenSSL 1.1.1d-freebsd 10 Sep 2019
linked to OpenSSL version: OpenSSL 1.1.1d-freebsd 10 Sep 2019
compiled with libxml2 version: 2.9.10
linked to libxml2 version: 20910
compiled with libjson-c version: 0.13.1
linked to libjson-c version: 0.13.1
compiled with zlib version: 1.2.11
linked to zlib version: 1.2.11
threads support is enabled
default paths:
named configuration: /usr/local/etc/namedb/named.conf
rndc configuration: /usr/local/etc/namedb/rndc.conf
DNSSEC root key: /usr/local/etc/namedb/bind.keys
nsupdate session key: /var/run/named/session.key
named PID file: /var/run/named/pid
named lock file: /var/run/named/named.lock
I upgraded bind to 9.11.20, and restarted it with -U 21000 .
After looking at the source, I added query-source and query-source-v6 in
the configuration even if the server has "recursion no;" and has no slave.
Today, same problem:
Jun 24 08:52:39 nsmaster named[46662]: general: error: could not get query source dispatcher (213.36.252.194#0)
Jun 24 08:52:39 nsmaster named[46662]: general: error: reloading configuration failed: out of memory
top show no memory issue:
last pid: 16247; load averages: 1.08, 1.80, 6.80 up 154+20:15:48 13:02:17
40 processes: 1 running, 39 sleeping
CPU: 1.6% user, 0.0% nice, 0.5% system, 0.1% interrupt, 97.8% idle
Mem: 4568M Active, 4978M Inact, 32G Wired, 580M Buf, 21G Free
ARC: 16G Total, 8811M MFU, 5160M MRU, 20M Anon, 471M Header, 1751M Other
12G Compressed, 38G Uncompressed, 3.31:1 Ratio
Swap: 64G Total, 64G Free
...
14164 bind 59 52 0 5320M 5200M sigwai 32 21.1H 0.31% named
...
% limits
Resource limits (current):
cputime infinity secs
filesize infinity kB
datasize 33554432 kB
stacksize 524288 kB
coredumpsize infinity kB
memoryuse infinity kB
memorylocked 64 kB
maxprocesses 63709
openfiles 1883583
sbsize infinity bytes
vmemoryuse infinity kB
pseudo-terminals infinity
swapuse infinity kB
kqueues infinity
umtxp infinity
# rndc status
version: BIND 9.11.20 (Extended Support Version) <id:f3d1d66>
running on nsmaster.free.org: FreeBSD amd64 12.1-RELEASE-p1 FreeBSD 12.1-RELEASE-p1 GENERIC
boot time: Wed, 24 Jun 2020 08:11:43 GMT
last configured: Wed, 24 Jun 2020 11:02:40 GMT
configuration file: /usr/local/etc/namedb/named-custom.conf
CPUs found: 56
worker threads: 56
UDP listeners per interface: 20
number of zones: 144218 (0 automatic)
debug level: 0
xfers running: 0
xfers deferred: 0
soa queries in progress: 0
query logging is ON
recursive clients: 0/900/1000
tcp clients: 6/1000
TCP high-water: 40
server is up and running
We have only about 70000 zones but bind semmes to count them twice when
they are signed.
I have rebuild bind with tuning=large :
# /usr/local/sbin/named -V
BIND 9.11.20 (Extended Support Version) <id:f3d1d66>
running on FreeBSD amd64 12.1-RELEASE-p1 FreeBSD 12.1-RELEASE-p1 GENERIC
built by make with '--localstatedir=/var' '--disable-linux-caps' '--with-randomdev=/dev/random' '--with-libxml2=/usr/local' '--with-readline=-L/usr/local/lib -ledit' '--with-dlopen=yes' '--with-gost=no' '--without-python' '--sysconfdir=/usr/local/etc/namedb' '--with-dlz-filesystem=yes' '--disable-dnstap' '--enable-filter-aaaa' '--disable-fixed-rrset' '--without-geoip2' '--without-gssapi' '--with-libidn2=/usr/local' '--enable-ipv6' '--with-libjson=/usr/local' '--disable-largefile' '--with-lmdb=/usr/local' '--disable-native-pkcs11' '--disable-querytrace' '--enable-rpz-nsdname' '--enable-rpz-nsip' 'STD_CDEFINES=-DDIG_SIGCHASE=1' '--with-openssl=/usr' '--enable-threads' '--with-tuning=large' '--disable-symtable' '--prefix=/usr/local' '--mandir=/usr/local/man' '--infodir=/usr/local/share/info/' '--build=amd64-portbld-freebsd12.1' 'build_alias=amd64-portbld-freebsd12.1' 'CC=cc' 'CFLAGS=-O2 -pipe -DLIBICONV_PLUG -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing ' 'LDFLAGS= -fstack-protector-strong ' 'LIBS=-L/usr/local/lib' 'CPPFLAGS=-DLIBICONV_PLUG -isystem /usr/local/include' 'CPP=cpp' 'PKG_CONFIG=pkgconf'
compiled by CLANG 4.2.1 Compatible FreeBSD Clang 8.0.1 (tags/RELEASE_801/final 366581)
compiled with OpenSSL version: OpenSSL 1.1.1d-freebsd 10 Sep 2019
linked to OpenSSL version: OpenSSL 1.1.1d-freebsd 10 Sep 2019
compiled with libxml2 version: 2.9.10
linked to libxml2 version: 20910
compiled with libjson-c version: 0.13.1
linked to libjson-c version: 0.13.1
compiled with zlib version: 1.2.11
linked to zlib version: 1.2.11
threads support is enabled
default paths:
named configuration: /usr/local/etc/namedb/named.conf
rndc configuration: /usr/local/etc/namedb/rndc.conf
DNSSEC root key: /usr/local/etc/namedb/bind.keys
nsupdate session key: /var/run/named/session.key
named PID file: /var/run/named/pid
named lock file: /var/run/named/named.lock
What can I do more to avoid this bug?
Is there a parameter or build option for such "big" server ?
Regards,
--
Laurent Frigault | Free.org - BookMyName.com - ONLINE SAS - Registar ID 74
More information about the bind-users
mailing list