Bind 9.16.28 upgrade: high memory utiization and OOM

[Ondřej Surý]

Hi Prasanna,

first of all, I would recommend you to conduct all the testing using the latest 9.16.33 release.

There is a difference in a memory usage between the versions, but you are not the first person to report the increased memory usage during the resolver operation. We've been unable to reproduce the issue locally.

However, you didn't send any details about the configuration, libraries used, operating system, etc.

I would suggest that you create a new issue in our gitlab instance, so we can track the information there: https://gitlab.isc.org/isc-projects/bind9/-/issues/new

Please summarize your findings, but also attach all relevant data: configuration used, compile time options used, how do you start named, etc.]

You should also gather the memory statistics from the named (XML or JSON) at datapoints, so we can look into differences.

For detailed memory debugging, I would recommend installing the latest jemalloc version and compiling BIND 9 with it (or `LD_LIBRARY_PRELOAD` it) and setup the heap profiling: https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling

I would either use lg_prof_interval for periodic dumps.

All the data points should be collected in the newly created GitLab issue.

Thanks,
Ondrej
--
Ondřej Surý (He/Him)
ondrej at isc.org

My working hours and your working hours may be different. Please do not feel obligated to reply outside your normal working hours.

> On 27. 9. 2022, at 16:09, Prasanna Mathivanan (pmathiva) via bind-users <bind-users at lists.isc.org> wrote:
> 
> Hi team,
> 
> We had recently upgraded our bind nameservers from 9.14.10 to 9.16.28. This led to the hosts gradually using up a lot of memory and eventually named was OOM killed as it consumed nearly 7GB out of total 8GB server memory. (This package was built from source for centos 7)
> 
> I’ve been looking into this and tested the performance of both 9.14 and 9.16 under the traffic of 600 queries per sec for 12 hours, which is the average qps our servers get. It was found that while 9.14 had a surge of around 2GB, 9.16 had a surge of 5.2GB during this time. I wanted to know whether this difference in memory consumption is expected while migrating from 9.14.10 to 9.16.28, or if this could be a memory leak that would keep building over time; it would really help if I can get some insights on what might be causing this, or if there’s any way to avoid this other ram bumping up the RAM.
> 
> Also I noticed some CVE related to this bind version recently, if anything to do with that ?
>  
> 	• A memory leak was fixed that could be externally triggered in the DNSSEC verification code for the ECDSA algorithm. (CVE-2022-38177) 
> 	• Memory leaks were fixed that could be externally triggered in the DNSSEC verification code for the EdDSA algorithm. (CVE-2022-38178)
>  
> I’d be glad to provide more info if needed. Would really appreciate your inputs and suggestions on this.
> -- 
> Regards,
> Prasanna.
> -- 
> Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list
> 
> ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.
> 
> 
> bind-users mailing list
> bind-users at lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users