High memory consumption in bind 9.18.2

Mon Aug 1 20:30:10 UTC 2022

Retesting with stock Debian bullseye (with extra OpenSSL 3.0.0)
and default configuration options:

TL;DR
9.16(git): 30648752
9.18(git): 30081280

So, pretty consistent with what I’ve seen here so far. And it seems that
the extra (developer) configure options, I’ve been using before have
little/no effect on the memory.

The 554MB less memory makes quite a sense because couple of bytes
related to MAP format has been removed from dns_rbtnode structure.

So, these are the results for just ./configure && make on Debian bullseye:

## BIND 9.16

01-Aug-2022 22:00:18.335 starting BIND 9.16.32-dev (Extended Support Version) <id:c863061a13>
01-Aug-2022 22:00:18.335 running on Linux x86_64 5.10.0-16-amd64 #1 SMP Debian 5.10.127-1 (2022-06-30)
01-Aug-2022 22:00:18.335 built with defaults
01-Aug-2022 22:00:18.335 running as: named -c named.conf -g -p 12345
01-Aug-2022 22:00:18.335 compiled by GCC 10.2.1 20210110
01-Aug-2022 22:00:18.335 compiled with OpenSSL version: OpenSSL 3.0.5 5 Jul 2022
01-Aug-2022 22:00:18.335 linked to OpenSSL version: OpenSSL 3.0.5 5 Jul 2022
01-Aug-2022 22:00:18.335 compiled with libxml2 version: 2.9.10
01-Aug-2022 22:00:18.335 linked to libxml2 version: 20910
01-Aug-2022 22:00:18.335 compiled with json-c version: 0.15
01-Aug-2022 22:00:18.335 linked to json-c version: 0.15
01-Aug-2022 22:00:18.335 compiled with zlib version: 1.2.11
01-Aug-2022 22:00:18.335 linked to zlib version: 1.2.11
01-Aug-2022 22:00:18.335 ----------------------------------------------------
01-Aug-2022 22:00:18.335 BIND 9 is maintained by Internet Systems Consortium,
01-Aug-2022 22:00:18.335 Inc. (ISC), a non-profit 501(c)(3) public-benefit
01-Aug-2022 22:00:18.335 corporation.  Support and training for BIND 9 are
01-Aug-2022 22:00:18.335 available at https://www.isc.org/support
01-Aug-2022 22:00:18.335 ----------------------------------------------------
01-Aug-2022 22:00:18.335 found 8 CPUs, using 8 worker threads
01-Aug-2022 22:00:18.335 using 8 UDP listeners per interface
01-Aug-2022 22:00:18.339 using up to 21000 sockets

Swap:0 USS:30648752 PSS:30649262 RSS:30652112

## BIND 9.18

01-Aug-2022 22:09:59.363 starting BIND 9.18.6-dev (Stable Release) <id:097a57e>
01-Aug-2022 22:09:59.363 running on Linux x86_64 5.10.0-16-amd64 #1 SMP Debian 5.10.127-1 (2022-06-30)
01-Aug-2022 22:09:59.363 built with default
01-Aug-2022 22:09:59.363 running as: named -c named.conf -g -p 12345
01-Aug-2022 22:09:59.363 compiled by GCC 10.2.1 20210110
01-Aug-2022 22:09:59.363 compiled with OpenSSL version: OpenSSL 3.0.5 5 Jul 2022
01-Aug-2022 22:09:59.363 linked to OpenSSL version: OpenSSL 3.0.5 5 Jul 2022
01-Aug-2022 22:09:59.363 compiled with libxml2 version: 2.9.10
01-Aug-2022 22:09:59.363 linked to libxml2 version: 20910
01-Aug-2022 22:09:59.363 compiled with json-c version: 0.15
01-Aug-2022 22:09:59.363 linked to json-c version: 0.15
01-Aug-2022 22:09:59.363 compiled with zlib version: 1.2.11
01-Aug-2022 22:09:59.363 linked to zlib version: 1.2.11
01-Aug-2022 22:09:59.363 ----------------------------------------------------
01-Aug-2022 22:09:59.363 BIND 9 is maintained by Internet Systems Consortium,
01-Aug-2022 22:09:59.363 Inc. (ISC), a non-profit 501(c)(3) public-benefit
01-Aug-2022 22:09:59.363 corporation.  Support and training for BIND 9 are
01-Aug-2022 22:09:59.363 available at https://www.isc.org/support
01-Aug-2022 22:09:59.363 ----------------------------------------------------
01-Aug-2022 22:09:59.363 found 8 CPUs, using 8 worker threads
01-Aug-2022 22:09:59.363 using 8 UDP listeners per interface

Swap:488612 USS:29592668 PSS:29593610 RSS:29596988

Ondrej
--
Ondřej Surý (He/Him)
ondrej at isc.org

My working hours and your working hours may be different. Please do not feel obligated to reply outside your normal working hours.

> On 1. 8. 2022, at 20:05, Dmitri Pavlov <dpavlov at perforce.com> wrote:
> 
> Thank you very much for the swift response, Ondrej,
> 
> You are right.  Local configuration is more "trivial" so to say -> it is RHEL 7.9 3.10.0-1160.71.1.el7.x86_64 with gcc-4.8.5-44.el7.x86_64.
> 
> Our installation / test flow was very basic. The records generation part is different though , using shell script basically.  Unique IPs were linked with unique CNAMEs.
> 
> The build process is very simple for all components: download -> configure -> make -> make install. All default values.
> 
> The only custom part is named.conf -> a single zone configured and custom hosts files. That is it.
> 
> We will test with a higher kernel.
> 
> Kind regards,
> Dmitri.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> -----Original Message-----
> From: Ondřej Surý <ondrej at isc.org>
> Sent: Monday, August 1, 2022 6:46 PM
> To: Dmitri Pavlov <dpavlov at perforce.com>
> Cc: bind-users at lists.isc.org
> Subject: Re: High memory consumption in bind 9.18.2
> 
>> On 1. 8. 2022, at 17:19, Dmitri Pavlov <dpavlov at perforce.com> wrote:
>> 
>> I’m pretty much sure you will get the same results in your lab.
> 
> I don’t want to delve into vague description of your experiment. You’ll have to share the exact scripts.
> 
> Even this:
> 
>> just the time-consuming task is to generate 100 MIL A + CNAME dummy records into the zone file.
> 
> could be violently different. Are those random names? Or deterministically generated? What are the CNAME targets?
> 
> What is the exact `named.conf`?
> 
> What are the exact library versions? What is the exact ./configure invocation?
> 
> How do you execute `named`?
> 
> And yet again - the memory statistics from the statschannel could show the breakdown of the internal memory contexts and also show if there’s a difference between the internally reported memory usage and externally reported memory usage, e.g. is this something allocated in named, or is this allocated in the libraries?
> 
>> Should the memory reduction apply to our experiment?
> 
> The question doesn’t really make sense.  We have not measured any increase in our test scenarios, which doesn’t mean you can’t find different scenarios with a memory increase.
> 
> Ondrej
> --
> Ondřej Surý (He/Him)
> ondrej at isc.org
> 
> My working hours and your working hours may be different. Please do not feel obligated to reply outside your normal working hours.
> 
> 
> 
> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
> 
> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
> 