High memory consumption in bind 9.18.2
Dmitri Pavlov
dpavlov at perforce.com
Tue Aug 2 15:21:18 UTC 2022
dnf install wget bzip2 gcc make -y
wget https://github.com/jemalloc/jemalloc/releases/download/5.3.0/jemalloc-5.3.0.tar.bz2
bzip2 -d jemalloc-5.3.0.tar.bz2 && tar -xf jemalloc-5.3.0.tar && cd jemalloc-5.3.0
./configure
make
make install
reboot -f
-----Original Message-----
From: Ondřej Surý <ondrej at isc.org>
Sent: Tuesday, August 2, 2022 6:20 PM
To: Dmitri Pavlov <dpavlov at perforce.com>
Cc: bind-users at lists.isc.org
Subject: Re: High memory consumption in bind 9.18.2
I don’t see jemalloc anywhere in your setup scripts. Preferably use the latest upstream jemalloc version available.
Ondřej
--
Ondřej Surý — ISC (He/Him)
My working hours and your working hours may be different. Please do not feel obligated to reply outside your normal working hours.
> On 2. 8. 2022, at 17:14, Dmitri Pavlov <dpavlov at perforce.com> wrote:
>
> Hi,
>
> Thank you very much for your feedback, Ondrej.
>
> Sharing the steps. Very simple: configure -> make -> make install. Very simple configuration. Just the zone file is big. Please, see the attached.
>
> 1. I followed the instructions from here
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbind
> 9.readthedocs.io%2Fen%2Fv9_18_4%2Fchapter10.html&data=05%7C01%7Cdp
> avlov%40perforce.com%7C0b7e44da128441c5ac7f08da749a8391%7C95b666d19a75
> 49ab95a38969fbcdc08c%7C0%7C0%7C637950504233087258%7CUnknown%7CTWFpbGZs
> b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D
> %7C3000%7C%7C%7C&sdata=v2cHf%2BQX%2FWMERTf%2F3H8HJYTXq2W6Y5mWjr%2B
> AedpXhho%3D&reserved=0
> No git.
>
> 2. Rocky 9
> uname -a
> Linux server.test.zone 5.14.0-70.17.1.el9_0.x86_64 #1 SMP PREEMPT Wed
> Jul 13 18:23:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux Package
> gcc-11.2.1-9.4.el9.x86_64
>
> 9.18.5
> ============= OUTPUT =============
> pmap -x 27821 | tail -n 1
> total kB 34853832 29425312 29416180
>
> smemstat -p named
> PID Swap USS PSS RSS User Command
> 27821 0.0 B 28.1 G 28.1 G 28.1 G root /usr/local/sbin/named
> Total: 0.0 B 28.1 G 28.1 G 28.1
> ============= ============= =============
>
> 9.16.21
> ============= OUTPUT =============
> pmap -x 18026 | tail -n 1
> total kB 28161152 27414164 27406208
>
> smemstat -p named
> PID Swap USS PSS RSS User Command
> 18026 0.0 B 26.1 G 26.1 G 26.1 G root /usr/local/sbin/named
> Total: 0.0 B 26.1 G 26.1 G 26.1 G
> ============= ============= =============
>
> 2 GB diff is still there.
>
> Steps have been attached.
>
> Will be very grateful, could you please, have a look at the steps provided? They are quite straightforward and self-explanatory. What is that "magic souse" I should apply, when there is Rocky 9 or RHEL 9 machine in front of me. I believe if I replicate your setup/steps, most likely I'll get the same output. However, if looking at the articles available to the wide audience (like this https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbind9.readthedocs.io%2Fen%2Fv9_18_4%2Fchapter10.html&data=05%7C01%7Cdpavlov%40perforce.com%7C0b7e44da128441c5ac7f08da749a8391%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C637950504233087258%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=v2cHf%2BQX%2FWMERTf%2F3H8HJYTXq2W6Y5mWjr%2BAedpXhho%3D&reserved=0 or any other(?)), what should my steps be to get the expected smaller memory footprint. Or maybe there are obvious flaws in the experiment?
>
> Thanking you very much for patience and cooperation, Dmitri.
>
>
>
>
> -----Original Message-----
> From: Ondřej Surý <ondrej at isc.org>
> Sent: Monday, August 1, 2022 8:18 PM
> To: Dmitri Pavlov <dpavlov at perforce.com>
> Cc: bind-users at lists.isc.org
> Subject: Re: High memory consumption in bind 9.18.2
>
> $ wc -l gen.db
> 100000000 gen.db
>
> generated with:
>
> #!/bin/env python3
>
> for x in range(0, 10000):
> for y in range(0, 2500):
> print(f"az{x}-{y} IN A 10.53.0.1")
> print(f"bz{x}-{y} IN A 10.53.0.2")
> print(f"ca{x}-{y} IN A 10.53.0.3")
> print(f"xh{x}-{y} IN CNAME az{x}-{y}”)
>
> ## BIND 9.16(git)
>
> 01-Aug-2022 19:06:27.151 starting BIND 9.16.32-dev (Extended Support Version) <id:c863061a13>
> 01-Aug-2022 19:06:27.151 running on Linux x86_64 5.10.0-16-amd64 #1 SMP Debian 5.10.127-1 (2022-06-30)
> 01-Aug-2022 19:06:27.151 built with 'CC=gcc-12' 'LD=' 'CFLAGS=-Ofast -ggdb -Wno-deprecated-declarations -fno-omit-frame-pointer -fno-optimize-sibling-calls -DISC_MEM_USE_INTERNAL_MALLOC=0 -DISC_MEM_TRACKLINES=1' 'LDFLAGS=' '--enable-developer' '--enable-warn-error' '--with-openssl' '--with-zlib' '--with-libxml2' '--with-json-c' '--with-readline' '--with-libidn2' '--enable-dnstap' '--with-libtool' '--without-make-clean' 'PKG_CONFIG_PATH=/home/ondrej/.local/lib/pkgconfig:'
> 01-Aug-2022 19:06:27.151 running as: named -c named.conf -g -p 12345
> 01-Aug-2022 19:06:27.151 compiled by GCC 12.0.1 20220319 (experimental) [master r12-7719-g8ca61ad148f]
> 01-Aug-2022 19:06:27.151 compiled with OpenSSL version: OpenSSL 3.0.5 5 Jul 2022
> 01-Aug-2022 19:06:27.151 linked to OpenSSL version: OpenSSL 3.0.5 5 Jul 2022
> 01-Aug-2022 19:06:27.151 compiled with libxml2 version: 2.9.10
> 01-Aug-2022 19:06:27.151 linked to libxml2 version: 20910
> 01-Aug-2022 19:06:27.151 compiled with json-c version: 0.15
> 01-Aug-2022 19:06:27.151 linked to json-c version: 0.15
> 01-Aug-2022 19:06:27.151 compiled with zlib version: 1.2.11
> 01-Aug-2022 19:06:27.151 linked to zlib version: 1.2.11
>
> $ smem -P name[d] -a
> SWAP:193304 USS:30451812 PSS:30452272 RSS:30454872
>
> ## BIND 9.18(git)
>
> 01-Aug-2022 18:12:57.227 starting BIND 9.18.6-dev (Stable Release) <id:097a57e>
> 01-Aug-2022 18:12:57.227 running on Linux x86_64 5.10.0-16-amd64 #1 SMP Debian 5.10.127-1 (2022-06-30)
> 01-Aug-2022 18:12:57.227 built with 'CC=gcc-12' 'LD=' 'CFLAGS=-Ofast -ggdb -Wno-deprecated-declarations -fno-omit-frame-pointer -fno-optimize-sibling-calls -DISC_MEM_USE_INTERNAL_MALLOC=0 -DISC_MEM_TRACKLINES=1' 'LDFLAGS=' '--enable-developer' '--enable-warn-error' '--with-openssl' '--with-zlib' '--with-libxml2' '--with-json-c' '--with-readline' '--with-libidn2' '--enable-dnstap' '--with-libtool' '--without-make-clean' 'PKG_CONFIG_PATH=/home/ondrej/.local/lib/pkgconfig:'
> 01-Aug-2022 18:12:57.227 running as: named -c named.conf -g -p 12345
> 01-Aug-2022 18:12:57.227 compiled by GCC 12.0.1 20220319 (experimental) [master r12-7719-g8ca61ad148f]
> 01-Aug-2022 18:12:57.227 compiled with OpenSSL version: OpenSSL 3.0.5 5 Jul 2022
> 01-Aug-2022 18:12:57.227 linked to OpenSSL version: OpenSSL 3.0.5 5 Jul 2022
> 01-Aug-2022 18:12:57.227 compiled with libxml2 version: 2.9.10
> 01-Aug-2022 18:12:57.227 linked to libxml2 version: 20910
> 01-Aug-2022 18:12:57.227 compiled with json-c version: 0.15
> 01-Aug-2022 18:12:57.227 linked to json-c version: 0.15
> 01-Aug-2022 18:12:57.227 compiled with zlib version: 1.2.11
> 01-Aug-2022 18:12:57.227 linked to zlib version: 1.2.11
>
> $ smem -P name[d] -a
> SWAP:645960 USS:29446788 PSS:29447529 RSS:29451056
>
> ## BIND 9.19(git)
>
> 01-Aug-2022 18:18:20.127 starting BIND 9.19.4-dev (Development Release) <id:fa1f35b>
> 01-Aug-2022 18:18:20.127 running on Linux x86_64 5.10.0-16-amd64 #1 SMP Debian 5.10.127-1 (2022-06-30)
> 01-Aug-2022 18:18:20.127 built with 'CC=gcc-12' 'LD=' 'CFLAGS=-Ofast -ggdb -Wno-deprecated-declarations -fno-omit-frame-pointer -fno-optimize-sibling-calls -DISC_MEM_USE_INTERNAL_MALLOC=0 -DISC_MEM_TRACKLINES=1' 'LDFLAGS=' '--enable-developer' '--enable-warn-error' '--with-openssl' '--with-zlib' '--with-libxml2' '--with-json-c' '--with-readline' '--with-libidn2' '--enable-dnstap' '--with-libtool' '--without-make-clean' 'PKG_CONFIG_PATH=/home/ondrej/.local/lib/pkgconfig:'
> 01-Aug-2022 18:18:20.127 running as: named -c named.conf -g -p 12345
> 01-Aug-2022 18:18:20.127 compiled by GCC 12.0.1 20220319 (experimental) [master r12-7719-g8ca61ad148f]
> 01-Aug-2022 18:18:20.127 compiled with OpenSSL version: OpenSSL 3.0.5 5 Jul 2022
> 01-Aug-2022 18:18:20.127 linked to OpenSSL version: OpenSSL 3.0.5 5 Jul 2022
> 01-Aug-2022 18:18:20.127 compiled with libxml2 version: 2.9.10
> 01-Aug-2022 18:18:20.127 linked to libxml2 version: 20910
> 01-Aug-2022 18:18:20.127 compiled with json-c version: 0.15
> 01-Aug-2022 18:18:20.127 linked to json-c version: 0.15
> 01-Aug-2022 18:18:20.127 compiled with zlib version: 1.2.11
> 01-Aug-2022 18:18:20.127 linked to zlib version: 1.2.11
>
> $ smem -P name[d] -a
> SWAP:1032108 USS:29062164 PSS:29062968 RSS:29066580
>
> So, it’s (USS+Swap):
> 9.16(git): 30645116
> 9.18(git): 30092748
> 9.19(git): 30094272
>
> So, no, I am unable to reproduce the results and yet again, I see a lower memory usage with 9.18+.
>
> But I see a common pattern here. I think both you and the OP were using CentOS/RHEL 7 which is using GCC 4.8.
>
> GCC was improved significantly since then. I would suggest to repeat the experiment on RHEL 9 if you can reproduce the same results.
>
> Ondrej
> --
> Ondřej Surý (He/Him)
> ondrej at isc.org
>
> My working hours and your working hours may be different. Please do not feel obligated to reply outside your normal working hours.
>
>>> On 1. 8. 2022, at 17:46, Ondřej Surý <ondrej at isc.org> wrote:
>>>
>>>> On 1. 8. 2022, at 17:19, Dmitri Pavlov <dpavlov at perforce.com> wrote:
>>>
>>> I’m pretty much sure you will get the same results in your lab.
>>
>> I don’t want to delve into vague description of your experiment.
>> You’ll have to share the exact scripts.
>>
>> Even this:
>>
>>> just the time-consuming task is to generate 100 MIL A + CNAME dummy records into the zone file.
>>
>> could be violently different. Are those random names? Or
>> deterministically generated? What are the CNAME targets?
>>
>> What is the exact `named.conf`?
>>
>> What are the exact library versions? What is the exact ./configure invocation?
>>
>> How do you execute `named`?
>>
>> And yet again - the memory statistics from the statschannel could show
>> the breakdown of the internal memory contexts and also show if there’s
>> a difference between the internally reported memory usage and
>> externally reported memory usage, e.g. is this something allocated in named, or is this allocated in the libraries?
>>
>>> Should the memory reduction apply to our experiment?
>>
>> The question doesn’t really make sense. We have not measured any
>> increase in our test scenarios, which doesn’t mean you can’t find different scenarios with a memory increase.
>>
>> Ondrej
>> --
>> Ondřej Surý (He/Him)
>> ondrej at isc.org
>>
>> My working hours and your working hours may be different. Please do not feel obligated to reply outside your normal working hours.
>>
>
>
>
> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>
> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>
CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
More information about the bind-users
mailing list