High memory consumption in bind 9.18.2

Tue Aug 2 15:29:52 UTC 2022

Well, then I don’t know the reason for the difference in your case. And I don’t personally see a compelling reason to investigate a 10% increase in artificial scenario like this since it apparently doesn’t apply to all scenarios. However, you are free to do the further investigation yourself.

We are refactoring the database for storing the resource records in 9.20 and it's probably better spent time to work on the refactoring than look at this. As usual, we would accept any well commented and well thought patches.

Ondřej
--
Ondřej Surý — ISC (He/Him)

My working hours and your working hours may be different. Please do not feel obligated to reply outside your normal working hours.

> On 2. 8. 2022, at 17:23, Dmitri Pavlov <dpavlov at perforce.com> wrote:
> 
> Hi,
> 
> Resending ... bad format.
> 
> dnf install wget bzip2 gcc make -y
> wget https://github.com/jemalloc/jemalloc/releases/download/5.3.0/jemalloc-5.3.0.tar.bz2
> bzip2 -d jemalloc-5.3.0.tar.bz2 && tar -xf jemalloc-5.3.0.tar && cd jemalloc-5.3.0
> ./configure
> make
> make install
> reboot -f
> 
> Dmitri.
> 
> 
> 
> -----Original Message-----
> From: Dmitri Pavlov
> Sent: Tuesday, August 2, 2022 6:14 PM
> To: Ondřej Surý <ondrej at isc.org>
> Cc: bind-users at lists.isc.org
> Subject: RE: High memory consumption in bind 9.18.2
> 
> Hi,
> 
> Thank you very much for your feedback, Ondrej.
> 
> Sharing the steps. Very simple: configure -> make -> make install. Very simple configuration. Just the zone file is big.  Please, see the attached.
> 
> 1. I followed the instructions from here https://bind9.readthedocs.io/en/v9_18_4/chapter10.html
> No git.
> 
> 2. Rocky 9
> uname -a
> Linux server.test.zone 5.14.0-70.17.1.el9_0.x86_64 #1 SMP PREEMPT Wed Jul 13 18:23:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux Package gcc-11.2.1-9.4.el9.x86_64
> 
> 9.18.5
> ============= OUTPUT =============
> pmap -x 27821 | tail -n 1
> total kB         34853832 29425312 29416180
> 
> smemstat -p named
>    PID      Swap       USS       PSS       RSS User       Command
>  27821     0.0 B    28.1 G    28.1 G    28.1 G root       /usr/local/sbin/named
> Total:      0.0 B    28.1 G    28.1 G    28.1
> ============= ============= =============
> 
> 9.16.21
> ============= OUTPUT =============
> pmap -x 18026 | tail -n 1
> total kB         28161152 27414164 27406208
> 
> smemstat -p named
>    PID      Swap       USS       PSS       RSS User       Command
>  18026     0.0 B    26.1 G    26.1 G    26.1 G root       /usr/local/sbin/named
> Total:      0.0 B    26.1 G    26.1 G    26.1 G
> ============= ============= =============
> 
> 2 GB diff is still there.
> 
> Steps have been attached.
> 
> Will be very grateful,  could you please, have a look at the steps provided? They are quite straightforward and self-explanatory. What is that "magic souse" I should apply, when there is Rocky 9 or RHEL 9 machine in front of me.  I believe if I replicate your setup/steps, most likely I'll get the same output. However, if looking at the articles available to the wide audience (like this https://bind9.readthedocs.io/en/v9_18_4/chapter10.html or any other(?)), what should my steps be to get the expected smaller memory footprint.  Or maybe there are obvious flaws in the experiment?
> 
> Thanking you very much for patience and cooperation, Dmitri.
> 
> 
> 
> 
> -----Original Message-----
> From: Ondřej Surý <ondrej at isc.org>
> Sent: Monday, August 1, 2022 8:18 PM
> To: Dmitri Pavlov <dpavlov at perforce.com>
> Cc: bind-users at lists.isc.org
> Subject: Re: High memory consumption in bind 9.18.2
> 
> $ wc -l gen.db
> 100000000 gen.db
> 
> generated with:
> 
> #!/bin/env python3
> 
> for x in range(0, 10000):
>    for y in range(0, 2500):
>        print(f"az{x}-{y} IN A 10.53.0.1")
>        print(f"bz{x}-{y} IN A 10.53.0.2")
>        print(f"ca{x}-{y} IN A 10.53.0.3")
>        print(f"xh{x}-{y} IN CNAME az{x}-{y}”)
> 
> ## BIND 9.16(git)
> 
> 01-Aug-2022 19:06:27.151 starting BIND 9.16.32-dev (Extended Support Version) <id:c863061a13>
> 01-Aug-2022 19:06:27.151 running on Linux x86_64 5.10.0-16-amd64 #1 SMP Debian 5.10.127-1 (2022-06-30)
> 01-Aug-2022 19:06:27.151 built with 'CC=gcc-12' 'LD=' 'CFLAGS=-Ofast -ggdb -Wno-deprecated-declarations -fno-omit-frame-pointer -fno-optimize-sibling-calls -DISC_MEM_USE_INTERNAL_MALLOC=0 -DISC_MEM_TRACKLINES=1' 'LDFLAGS=' '--enable-developer' '--enable-warn-error' '--with-openssl' '--with-zlib' '--with-libxml2' '--with-json-c' '--with-readline' '--with-libidn2' '--enable-dnstap' '--with-libtool' '--without-make-clean' 'PKG_CONFIG_PATH=/home/ondrej/.local/lib/pkgconfig:'
> 01-Aug-2022 19:06:27.151 running as: named -c named.conf -g -p 12345
> 01-Aug-2022 19:06:27.151 compiled by GCC 12.0.1 20220319 (experimental) [master r12-7719-g8ca61ad148f]
> 01-Aug-2022 19:06:27.151 compiled with OpenSSL version: OpenSSL 3.0.5 5 Jul 2022
> 01-Aug-2022 19:06:27.151 linked to OpenSSL version: OpenSSL 3.0.5 5 Jul 2022
> 01-Aug-2022 19:06:27.151 compiled with libxml2 version: 2.9.10
> 01-Aug-2022 19:06:27.151 linked to libxml2 version: 20910
> 01-Aug-2022 19:06:27.151 compiled with json-c version: 0.15
> 01-Aug-2022 19:06:27.151 linked to json-c version: 0.15
> 01-Aug-2022 19:06:27.151 compiled with zlib version: 1.2.11
> 01-Aug-2022 19:06:27.151 linked to zlib version: 1.2.11
> 
> $ smem -P name[d] -a
> SWAP:193304 USS:30451812 PSS:30452272 RSS:30454872
> 
> ## BIND 9.18(git)
> 
> 01-Aug-2022 18:12:57.227 starting BIND 9.18.6-dev (Stable Release) <id:097a57e>
> 01-Aug-2022 18:12:57.227 running on Linux x86_64 5.10.0-16-amd64 #1 SMP Debian 5.10.127-1 (2022-06-30)
> 01-Aug-2022 18:12:57.227 built with  'CC=gcc-12' 'LD=' 'CFLAGS=-Ofast  -ggdb -Wno-deprecated-declarations -fno-omit-frame-pointer -fno-optimize-sibling-calls -DISC_MEM_USE_INTERNAL_MALLOC=0 -DISC_MEM_TRACKLINES=1' 'LDFLAGS=' '--enable-developer' '--enable-warn-error' '--with-openssl' '--with-zlib' '--with-libxml2' '--with-json-c' '--with-readline' '--with-libidn2' '--enable-dnstap' '--with-libtool' '--without-make-clean' 'PKG_CONFIG_PATH=/home/ondrej/.local/lib/pkgconfig:'
> 01-Aug-2022 18:12:57.227 running as: named -c named.conf -g -p 12345
> 01-Aug-2022 18:12:57.227 compiled by GCC 12.0.1 20220319 (experimental) [master r12-7719-g8ca61ad148f]
> 01-Aug-2022 18:12:57.227 compiled with OpenSSL version: OpenSSL 3.0.5 5 Jul 2022
> 01-Aug-2022 18:12:57.227 linked to OpenSSL version: OpenSSL 3.0.5 5 Jul 2022
> 01-Aug-2022 18:12:57.227 compiled with libxml2 version: 2.9.10
> 01-Aug-2022 18:12:57.227 linked to libxml2 version: 20910
> 01-Aug-2022 18:12:57.227 compiled with json-c version: 0.15
> 01-Aug-2022 18:12:57.227 linked to json-c version: 0.15
> 01-Aug-2022 18:12:57.227 compiled with zlib version: 1.2.11
> 01-Aug-2022 18:12:57.227 linked to zlib version: 1.2.11
> 
> $ smem -P name[d] -a
> SWAP:645960 USS:29446788 PSS:29447529 RSS:29451056
> 
> ## BIND 9.19(git)
> 
> 01-Aug-2022 18:18:20.127 starting BIND 9.19.4-dev (Development Release) <id:fa1f35b>
> 01-Aug-2022 18:18:20.127 running on Linux x86_64 5.10.0-16-amd64 #1 SMP Debian 5.10.127-1 (2022-06-30)
> 01-Aug-2022 18:18:20.127 built with  'CC=gcc-12' 'LD=' 'CFLAGS=-Ofast  -ggdb -Wno-deprecated-declarations -fno-omit-frame-pointer -fno-optimize-sibling-calls -DISC_MEM_USE_INTERNAL_MALLOC=0 -DISC_MEM_TRACKLINES=1' 'LDFLAGS=' '--enable-developer' '--enable-warn-error' '--with-openssl' '--with-zlib' '--with-libxml2' '--with-json-c' '--with-readline' '--with-libidn2' '--enable-dnstap' '--with-libtool' '--without-make-clean' 'PKG_CONFIG_PATH=/home/ondrej/.local/lib/pkgconfig:'
> 01-Aug-2022 18:18:20.127 running as: named -c named.conf -g -p 12345
> 01-Aug-2022 18:18:20.127 compiled by GCC 12.0.1 20220319 (experimental) [master r12-7719-g8ca61ad148f]
> 01-Aug-2022 18:18:20.127 compiled with OpenSSL version: OpenSSL 3.0.5 5 Jul 2022
> 01-Aug-2022 18:18:20.127 linked to OpenSSL version: OpenSSL 3.0.5 5 Jul 2022
> 01-Aug-2022 18:18:20.127 compiled with libxml2 version: 2.9.10
> 01-Aug-2022 18:18:20.127 linked to libxml2 version: 20910
> 01-Aug-2022 18:18:20.127 compiled with json-c version: 0.15
> 01-Aug-2022 18:18:20.127 linked to json-c version: 0.15
> 01-Aug-2022 18:18:20.127 compiled with zlib version: 1.2.11
> 01-Aug-2022 18:18:20.127 linked to zlib version: 1.2.11
> 
> $ smem -P name[d] -a
> SWAP:1032108 USS:29062164 PSS:29062968 RSS:29066580
> 
> So, it’s (USS+Swap):
> 9.16(git): 30645116
> 9.18(git): 30092748
> 9.19(git): 30094272
> 
> So, no, I am unable to reproduce the results and yet again, I see a lower memory usage with 9.18+.
> 
> But I see a common pattern here. I think both you and the OP were using CentOS/RHEL 7 which is using GCC 4.8.
> 
> GCC was improved significantly since then. I would suggest to repeat the experiment on RHEL 9 if you can reproduce the same results.
> 
> Ondrej
> --
> Ondřej Surý (He/Him)
> ondrej at isc.org
> 
> My working hours and your working hours may be different. Please do not feel obligated to reply outside your normal working hours.
> 
>>> On 1. 8. 2022, at 17:46, Ondřej Surý <ondrej at isc.org> wrote:
>>> 
>>>> On 1. 8. 2022, at 17:19, Dmitri Pavlov <dpavlov at perforce.com> wrote:
>>> 
>>> I’m pretty much sure you will get the same results in your lab.
>> 
>> I don’t want to delve into vague description of your experiment.
>> You’ll have to share the exact scripts.
>> 
>> Even this:
>> 
>>> just the time-consuming task is to generate 100 MIL A + CNAME dummy records into the zone file.
>> 
>> could be violently different. Are those random names? Or
>> deterministically generated? What are the CNAME targets?
>> 
>> What is the exact `named.conf`?
>> 
>> What are the exact library versions? What is the exact ./configure invocation?
>> 
>> How do you execute `named`?
>> 
>> And yet again - the memory statistics from the statschannel could show
>> the breakdown of the internal memory contexts and also show if there’s
>> a difference between the internally reported memory usage and
>> externally reported memory usage, e.g. is this something allocated in named, or is this allocated in the libraries?
>> 
>>> Should the memory reduction apply to our experiment?
>> 
>> The question doesn’t really make sense.  We have not measured any
>> increase in our test scenarios, which doesn’t mean you can’t find different scenarios with a memory increase.
>> 
>> Ondrej
>> --
>> Ondřej Surý (He/Him)
>> ondrej at isc.org
>> 
>> My working hours and your working hours may be different. Please do not feel obligated to reply outside your normal working hours.
>> 
> 
> 
> 
> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
> 
> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
> 