dnsperf and BIND memory consumption

Thu Dec 11 08:25:42 UTC 2008

OK. I just make bind from src with ./configure --enable-threads & gcc
option -static.

file /usr/local/sbin/named-test
/usr/local/sbin/named-test: ELF 64-bit LSB executable, x86-64, version 1
(FreeBSD), for FreeBSD 7.1 (701100), statically linked, FreeBSD-style,
not stripped

fresh system (yesterday cvsup to RELENG_7)
$ uname -a
FreeBSD XXX.XXX.XX 7.1-PRERELEASE FreeBSD 7.1-PRERELEASE #1: Wed Dec 10
17:07:03 MSK 2008     root at XXXX:/usr/obj/usr/src/sys/XXX  amd64

2.
  max-cache-size 128M;

 start:
 /usr/bin/limits -c 1000M -v 500M /usr/local/sbin/named-test -c
/etc/namedb/named.conf

$ gdb -c named-test.core -se /usr/local/sbin/named-test
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Core was generated by `named-test'.
Program terminated with signal 6, Aborted.
#0  0x000000000058c3fc in thr_kill ()
[New Thread 0x80902f00 (LWP 100404)]
[New Thread 0x80902d80 (LWP 100400)]
[New Thread 0x80902c00 (LWP 100356)]
[New Thread 0x80902a80 (LWP 100318)]
[New Thread 0x80902900 (LWP 100239)]
[New Thread 0x80902780 (LWP 100237)]
[New Thread 0x80902600 (LWP 100222)]
[New Thread 0x80902480 (LWP 100209)]
[New Thread 0x80902300 (LWP 100175)]
[New Thread 0x80902180 (LWP 100092)]
[New Thread 0x80803180 (LWP 100177)]
(gdb) bt
#0  0x000000000058c3fc in thr_kill ()
#1  0x00000000005c5a68 in abort ()
#2  0x0000000000597af7 in malloc ()
#3  0x0000000000564247 in mem_getunlocked (ctx=0x8080d140, size=94) at
mem.c:385
#4  0x0000000000564b68 in isc__mem_get (ctx=0x8080d140, size=94,
file=0x61bd78 "rbt.c", line=1425) at mem.c:1096
#5  0x0000000000480121 in create_node (mctx=0x8080d140,
name=0x7fffffbfcff0, nodep=0x7fffffbfd2e0) at rbt.c:1424
#6  0x000000000048080f in dns_rbt_addnode (rbt=0x80a925e8,
name=0x88cf2020, nodep=0x7fffffbfd3a8) at rbt.c:624
#7  0x000000000048be53 in loading_addrdataset (arg=0x94b07ff0,
name=0x88cf2020, rdataset=0x7fffffbfd810) at rbtdb.c:5657
#8  0x0000000000463761 in commit (callbacks=0x7fffffbfe5c0,
lctx=0x80834000, head=0x7fffffbfe480, owner=0x88cf2020,
    source=0x94c2afd8 "co/brand.bak", line=611215) at master.c:2729
#9  0x00000000004668df in load_text (lctx=0x80834000) at master.c:1427
#10 0x000000000046b61b in dns_master_loadfile2 (master_file=0x878a7098
"co/broad.bak", top=Variable "top" is not available.
)
    at master.c:2350
#11 0x0000000000506126 in zone_load (zone=0x878ec000, flags=Variable
"flags" is not available.
) at zone.c:1504
#12 0x00000000005082b9 in load (zone=Variable "zone" is not available.
) at zt.c:246
#13 0x0000000000507ec2 in dns_zt_apply2 (zt=Variable "zt" is not available.
) at zt.c:379
#14 0x0000000000508144 in dns_zt_load (zt=0x86adb750,
stop=isc_boolean_false) at zt.c:237
#15 0x00000000004223c7 in load_zones (server=0x8082f000,
stop=isc_boolean_false) at server.c:3659
#16 0x00000000004232fc in run_server (task=Variable "task" is not available.
) at server.c:3751
#17 0x000000000057052c in run (uap=Variable "uap" is not available.
) at task.c:862
#18 0x00000000005868a7 in thread_start ()
#19 0x0000000000000000 in ?? ()
Cannot access memory at address 0x7fffffbff000

At normal situation after startup memory usage over 700 MB.

-------------

JINMEI Tatuya / 神明達哉 wrote:
> At Wed, 10 Dec 2008 15:50:22 +0300,
> Dmitry Rybin <kirgudu at corbina.net> wrote:
>> JINMEI Tatuya / 神明達哉 wrote:
>>> At Tue, 09 Dec 2008 18:05:27 +0300,
>>> Dmitry Rybin <kirgudu at corbina.net> wrote:
>>>
>>>> I test patch, add to bind95/Makefile
>>>> .if (${ARCH} == "amd64")
>>>> ARCH=           x86_64
>>>> .endif
>>> Future versions of BIND9 will support amd64 in its configure script to
>>> workaround the FreeBSD port for amd64.
>>>
>>> Regarding the memory leak, I believe it's already solved in 9.5.1rc1
>>> (even with threads and without atomic).
>> I just make port bind 9.5.1rc1. It has same problem with memory leak.
>> It grows from 670M on startup, to 1,4Gb after 20 minutes of work.
> 
> Can you first fall back to the vanilla 9.5.1rc1 (i.e., not FreeBSD
> port) so that we can separate FreeBSD-port specific issue and BIND9
> specific leak?
> 
> Second, what if you stop named by 'rndc stop'?  If there's memory leak
> in BIND9, it normally detects it during a cleanup process and
> indicates the bug by aborting (core dumping) itself.
> 
> If it doesn't cause an abort, please then try the diagnosing I
> suggested before:
> http://marc.info/?l=bind-users&m=121811979629090&w=2
>  
> To summarize it:
> 
> 1. create a symbolic link from "/etc/malloc.conf" to "X":
>    # ln -s X /etc/malloc.conf
> 2. - start named with a moderate limitation of virtual memory size, e.g.
>    # /usr/bin/limits -v 384m $path_to_named/named <command line options>
> (note that "384m" should be reasonably large compared with
> max-cache-size.  I'd suggest setting max-cache-size to 128M and
> setting 'limits -v' to 512m).
> 3. Then the named process will eventually abort itself with a core dump
>    due to malloc failure.  Please show us the stack trace at that point.
>    Hopefully it will reveal the malloc call that keeps consuming memory.
> 
> In fact, I myself successfully identified one leak in 9.5.0-P2 with
> FreeBSD port this way.
> 
> ---
> JINMEI, Tatuya
> Internet Systems Consortium, Inc.
> 