High memory consumption in bind 9.18.2

Raman kumar kumarraman.mca at gmail.com
Mon Jul 11 04:09:26 UTC 2022 

Hello,

Did you get a chance to look into the data that I shared above? The
tools(ps_mem, pmap ) that you recommended to compare the RAM consumption
between 9.18.x and 9.16.x are also showing that memory consumption is more
in 9.18.x compared to 9.16.x.

Thanks in advance.

Regards,
Raman




On Fri, Jul 1, 2022 at 9:08 PM Raman kumar <kumarraman.mca at gmail.com> wrote:

> Hello,
>
> In continuation to my previous mails, sharing memory usage by bind 9.16.21
> and 9.18.3 using pmap and ps_mem tools.
>
> We have observed memory usage in 9.18.3 is higher by approximately 1.10 GB
> with the same amount of data loaded on both bind versions.
> Attaching pmap output for both bind versions in this mail which also shows
> memory used by heap/mmap on 9.18.3 is ~10.16GB whereas 9.16.21 was
> consuming ~9.05GB
>
> 9.16.21
> ======
> ps_mem 7543
> Private  +   Shared  =  RAM used Program
> 9.1 GiB + 262.5 KiB =   9.1 GiB named
>
> 9.18.3
> =====
> ps_mem 8456
> Private  +   Shared  =  RAM used Program
> 10.2 GiB + 162.0 KiB =  10.2 GiB named
>
> Regards,
> Raman
>
> On Fri, Jun 24, 2022 at 10:06 AM Ondřej Surý <ondrej at isc.org> wrote:
>
>>
>> On 24. 6. 2022, at 5:28, Nagaraju Amarana <amarana.nagaraju at gmail.com>
>> wrote:
>>
>> 
>> Hi Ondrej,
>>
>> As per the BIND stats usage is lower than the 9.16 however it is in the
>> other way when compared with the system stats. Any idea on this difference?
>>
>>
>> Hi,
>>
>> I already mentioned some below - memory fragmentation plays a role. Using
>> jemalloc helps with that.
>>
>> Other factors are mentioned in the resources I’ve sent - have you read
>> it? - shared libraries are also accounted in this space. Using the numbers
>> from statm (or even ps/top) is only informative and makes sense only as
>> hint - f.e. when memory leaks. Or comparing the exactly same builds with
>> same libraries linked etc.
>>
>> I think user need to consider the memory allocation based on the system
>> usage otherwise OS may crash the process due to out of memory.
>>
>>
>> No, not really. First of all, OOM killer makes sure it doesn’t crash the
>> operating system, but kills a selected process to free up the memory.
>>
>> Do you agree? With this can i say 9.18.x memory usage is higher than
>> 9.16.x?
>>
>>
>> No, not really.
>>
>> Ondrej
>> --
>> Ondřej Surý — ISC (He/Him)
>>
>> My working hours and your working hours may be different. Please do not
>> feel obligated to reply outside your normal working hours.
>>
>> Regards,
>> Nagaraju A.
>>
>>
>> On Fri, Jun 24, 2022 at 12:08 AM Ondřej Surý <ondrej at isc.org> wrote:
>>
>>> The stats that you sent does show that 9.18 is internally using little
>>> bit less memory than 9.16. Also please note that your 9.16 version is
>>> outdated.
>>>
>>> As for the differences - are you using jemalloc to compile BIND 9.18?
>>> This is the recommended memory allocator for production as it does greatly
>>> reduce memory fragmentation and increase performance. The statm output is
>>> not 100% accurate, you need to analyze the output of smaps file to get an
>>> accurate picture, I’ve listed some tools that can do that below.
>>>
>>> Measuring actual memory usage is difficult, there are some good hints
>>> here:
>>> https://stackoverflow.com/questions/131303/how-can-i-measure-the-actual-memory-usage-of-an-application-or-process
>>>
>>> That said, I see nothing wrong in the output you posted here. In fact,
>>> what Petr said earlier is true - the memory usage went down. (Just
>>> disregard the “total” from stats - it’s misleading it never goes down, just
>>> up, it only accounts allocations, but deallocations are not counted - e.g.
>>> it's counter only for developers.)
>>>
>>> Use pmap, ps_mem, smem or valgrind massif tool. You can also make
>>> jemalloc to print various memory statistics at the program exit by setting
>>> environment variable (see the man page for jemalloc for more details).
>>>
>>> Ondřej
>>> --
>>> Ondřej Surý — ISC (He/Him)
>>>
>>> My working hours and your working hours may be different. Please do not
>>> feel obligated to reply outside your normal working hours.
>>>
>>> On 23. 6. 2022, at 19:07, Raman kumar <kumarraman.mca at gmail.com> wrote:
>>>
>>> 
>>> Hello,
>>>
>>> Thanks for your reply!
>>>
>>> I did the analysis of stats generated, top command output and statm
>>> file. Memory consumption(inUse) in 9.16 is matching in all places.
>>>
>>> In 9.18 memory consumption(inUse) is matching in statm file and top
>>> command output, but it is not matching with stats generated (please see
>>> images below for stats, top command output and statm of bind version 9.16
>>> and 9.18).
>>>
>>> Can you please guide us to debug high memory uses in bind 9.18 based
>>> upon the data shared below?
>>> Also need your inputs why stats are not matching with memory(InUse) in
>>> bind version 9.18?
>>>
>>> [image: image.png]
>>>
>>> [image: image.png]
>>> [image: image.png]
>>> Regards,
>>> Raman
>>>
>>> On Wed, Jun 22, 2022 at 10:54 AM Ondřej Surý <ondrej at isc.org> wrote:
>>>
>>>> And what did you find looking at the new data? What are the
>>>> differences? And by how much?
>>>>
>>>> You should not expect other people doing the analysis for yourself.
>>>>
>>>> Ondrej
>>>> --
>>>> Ondřej Surý — ISC (He/Him)
>>>>
>>>> My working hours and your working hours may be different. Please do not
>>>> feel obligated to reply outside your normal working hours.
>>>>
>>>> On 22. 6. 2022, at 6:30, Raman kumar <kumarraman.mca at gmail.com> wrote:
>>>>
>>>> 
>>>> Hello,
>>>>
>>>> As suggested, please find the attached artifacts which includes stats
>>>> by configuring statistics channel in named.conf, content at
>>>> /proc/<PID>/statm, top command output for both 9.16.21 and 9.18.3.
>>>>
>>>> Regards,
>>>> Raman
>>>>
>>>> On Thu, Jun 16, 2022 at 12:55 PM Ondřej Surý <ondrej at isc.org> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> we cannot really help you if you ignore everything that was said to
>>>>> you regarding the memory measurements.
>>>>>
>>>>> Ondrej.
>>>>> --
>>>>> Ondřej Surý (He/Him)
>>>>> ondrej at isc.org
>>>>>
>>>>> My working hours and your working hours may be different. Please do
>>>>> not feel obligated to reply outside your normal working hours.
>>>>>
>>>>> > On 16. 6. 2022, at 9:02, Raman kumar <kumarraman.mca at gmail.com>
>>>>> wrote:
>>>>> >
>>>>> > Hello All,
>>>>> >
>>>>> > We configured bind 9.18, using jemalloc but still memory consumption
>>>>> is high in 9.18 as compared to 9.16.
>>>>> >
>>>>> > On version 9.16.21, RAM consumption was 3.8 GB without jemalloc. And
>>>>> on 9.18.2, RAM consumption is 4.2 GB with jemalloc with the same data.
>>>>> >
>>>>> > Is this the expected behaviour or any more tuning is needed?
>>>>> >
>>>>> > One more thing: does CNAME record length also impact the memory used?
>>>>> >
>>>>> > Regards,
>>>>> > Raman
>>>>> >
>>>>> >
>>>>> > On Thu, May 19, 2022 at 3:52 PM Petr Špaček <pspacek at isc.org> wrote:
>>>>> > On 18. 05. 22 22:39, Ondřej Surý wrote:
>>>>> > > Hi Klarstein,
>>>>> > >
>>>>> > > Gathering the output of named statschannel should be good enough
>>>>> for initial assessment (json please).
>>>>> > >
>>>>> > > For 9.18, make sure the jemalloc is being used at runtime.
>>>>> >
>>>>> > Here are commands you asked for:
>>>>> >
>>>>> > 1. when running ./configure, make sure the output at the end has
>>>>> this:
>>>>> >
>>>>> > Configuration summary:
>>>>> >
>>>>> -------------------------------------------------------------------------------
>>>>> > Optional features enabled:
>>>>> >      Memory allocator: jemalloc
>>>>> >
>>>>> >
>>>>> > 2. Then, configure statistics channel in named.conf like this:
>>>>> >
>>>>> > statistics-channels {
>>>>> >         inet 127.0.0.1 port 8080;
>>>>> > };
>>>>> >
>>>>> >
>>>>> > 3. With that in place you can grab stats from this URL:
>>>>> > http://127.0.0.1:8080/json/v1
>>>>> >
>>>>> > Configuration for v9.16 is the same, just skip the jemalloc part.
>>>>> >
>>>>> > 4. Bonus points for grabbing /proc/<PID>/statm content at the same
>>>>> time
>>>>> > as content of the JSON stats endpoint (if you are on Linux).
>>>>> >
>>>>> > I hope it helps.
>>>>> > Petr Špaček
>>>>> >
>>>>> >
>>>>> >
>>>>> > >
>>>>> > > Ondrej
>>>>> > > --
>>>>> > > Ondřej Surý — ISC (He/Him)
>>>>> > >
>>>>> > > My working hours and your working hours may be different. Please
>>>>> do not feel obligated to reply outside your normal working hours.
>>>>> > >
>>>>> > >> On 18. 5. 2022, at 22:32, Klaus Darilion via bind-users <
>>>>> bind-users at lists.isc.org> wrote:
>>>>> > >>
>>>>> > >> Can you please provide some commands whose output you are
>>>>> interested? I want to collect the statistics for 9.16 before updating to
>>>>> 9.18.
>>>>> > >> Thanks
>>>>> > >> Klaus
>>>>> > >>
>>>>> > >>> -----Ursprüngliche Nachricht-----
>>>>> > >>> Von: bind-users <bind-users-bounces at lists.isc.org> Im Auftrag
>>>>> von Petr
>>>>> > >>> Špacek
>>>>> > >>> Gesendet: Mittwoch, 18. Mai 2022 18:20
>>>>> > >>> An: bind-users at lists.isc.org
>>>>> > >>> Betreff: Re: AW: High memory consumption in bind 9.18.2
>>>>> > >>>
>>>>> > >>> I would be very interested in hearing more!
>>>>> > >>>
>>>>> > >>> In majority of our internal testing 9.16 has higher memory
>>>>> consumption
>>>>> > >>> than 9.18, especially when 9.18 is compiled with libjemalloc.
>>>>> And the
>>>>> > >>> differences are not small, for some configurations it can be
>>>>> even 2x or
>>>>> > >>> 3x more on 9.16 than it is on 9.18.
>>>>> > >>>
>>>>> > >>> If you encounter it again please get back to us so we can
>>>>> diagnose it.
>>>>> > >>>
>>>>> > >>> Thank you!
>>>>> > >>> Petr Špaček
>>>>> > >>>
>>>>> > >>>
>>>>> > >>>> On 18. 05. 22 8:56, Klaus Darilion via bind-users wrote:
>>>>> > >>>> I remember we had similar issues with 9.18 (isc ppa packages)
>>>>> and hence
>>>>> > >>> wen't back to 9.16. But I can not remember the details.
>>>>> > >>>>
>>>>> > >>>> regards
>>>>> > >>>> Klaus
>>>>> > >>>>
>>>>> > >>>>> -----Ursprüngliche Nachricht-----
>>>>> > >>>>> Von: bind-users <bind-users-bounces at lists.isc.org> Im Auftrag
>>>>> von
>>>>> > >>> Ondrej
>>>>> > >>>>> Surý101 71 l t1h, 18. Mai 2022 08:37
>>>>> > >>>>> An: Raman kumar <kumarraman.mca at gmail.com>
>>>>> > >>>>> Cc: bind-users at lists.isc.org
>>>>> > >>>>> Betreff: Re: High memory consumption in bind 9.18.2
>>>>> > >>>>>
>>>>> > >>>>> You did not provided any details, so we can’t really help you.
>>>>> > >>>>>
>>>>> > >>>>> What is “RAM consumption” anyway? VSZ, RSS, numbers pulled from
>>>>> > >>> stats
>>>>> > >>>>> channel from named?
>>>>> > >>>>>
>>>>> > >>>>> What’s the hardware, what is the configuration, how was BIND 9
>>>>> compiled
>>>>> > >>>>> (or packaged)?
>>>>> > >>>>>
>>>>> > >>>>> The more details, the better
>>>>> > >>>>>
>>>>> > >>>>> Ondrej
>>>>> > >>>>> --
>>>>> > >>>>> Ondřej Surý (He/Him)
>>>>> > >>>>> ondrej at isc.org
>>>>> > >>>>>
>>>>> > >>>>> My working hours and your working hours may be different.
>>>>> Please do
>>>>> > >>> not
>>>>> > >>>>> feel obligated to reply outside your normal working hours.
>>>>> > >>>>>
>>>>> > >>>>>> On 18. 5. 2022, at 8:32, Raman kumar <
>>>>> kumarraman.mca at gmail.com>
>>>>> > >>>>> wrote:
>>>>> > >>>>>>
>>>>> > >>>>>> Hello Team,
>>>>> > >>>>>>
>>>>> > >>>>>> While upgrading from BIND 9.16.10 to 9.18.2, we have observed
>>>>> high
>>>>> > >>>>> memory consumption.
>>>>> > >>>>>>
>>>>> > >>>>>> On version 9.16.2, RAM consumption was 3.8 GB. And on 9.18.2,
>>>>> RAM
>>>>> > >>>>> consumption is 4.5 GB. Due to this an increase of
>>>>> approximately 20 %
>>>>> > >>>>> memory is observed.
>>>>> > >>>>>>
>>>>> > >>>>>> Is this the expected behaviour or any tuning is needed?
>>>>> > >>>>>>
>>>>> > >>>>>> Thanks in advance.
>>>>> > >>>>>>
>>>>> > >>>>>> Regards,
>>>>> > >>>>>> Raman
>>>>> > >>>>>> --
>>>>> > >>>>>> Visit https://lists.isc.org/mailman/listinfo/bind-users to
>>>>> unsubscribe
>>>>> > >>> from
>>>>> > >>>>> this list
>>>>> > >>>>>>
>>>>> > >>>>>> ISC funds the development of this software with paid support
>>>>> > >>>>> subscriptions. Contact us at https://www.isc.org/contact/ for
>>>>> more
>>>>> > >>>>> information.
>>>>> > >>>>>>
>>>>> > >>>>>>
>>>>> > >>>>>> bind-users mailing list
>>>>> > >>>>>> bind-users at lists.isc.org
>>>>> > >>>>>> https://lists.isc.org/mailman/listinfo/bind-users
>>>>> > >>>>
>>>>> > >>>
>>>>> > >>>
>>>>> > >>> --
>>>>> > >>> Petr Špaček
>>>>> > >>> --
>>>>> > >>> Visit https://lists.isc.org/mailman/listinfo/bind-users to
>>>>> unsubscribe from this
>>>>> > >>> list
>>>>> > >>>
>>>>> > >>> ISC funds the development of this software with paid support
>>>>> subscriptions.
>>>>> > >>> Contact us at https://www.isc.org/contact/ for more information.
>>>>> > >>>
>>>>> > >>>
>>>>> > >>> bind-users mailing list
>>>>> > >>> bind-users at lists.isc.org
>>>>> > >>> https://lists.isc.org/mailman/listinfo/bind-users
>>>>> > >> --
>>>>> > >> Visit https://lists.isc.org/mailman/listinfo/bind-users to
>>>>> unsubscribe from this list
>>>>> > >>
>>>>> > >> ISC funds the development of this software with paid support
>>>>> subscriptions. Contact us at https://www.isc.org/contact/ for more
>>>>> information.
>>>>> > >>
>>>>> > >>
>>>>> > >> bind-users mailing list
>>>>> > >> bind-users at lists.isc.org
>>>>> > >> https://lists.isc.org/mailman/listinfo/bind-users
>>>>> >
>>>>> >
>>>>> > --
>>>>> > Petr Špaček
>>>>> > --
>>>>> > Visit https://lists.isc.org/mailman/listinfo/bind-users to
>>>>> unsubscribe from this list
>>>>> >
>>>>> > ISC funds the development of this software with paid support
>>>>> subscriptions. Contact us at https://www.isc.org/contact/ for more
>>>>> information.
>>>>> >
>>>>> >
>>>>> > bind-users mailing list
>>>>> > bind-users at lists.isc.org
>>>>> > https://lists.isc.org/mailman/listinfo/bind-users
>>>>> > --
>>>>> > Visit https://lists.isc.org/mailman/listinfo/bind-users to
>>>>> unsubscribe from this list
>>>>> >
>>>>> > ISC funds the development of this software with paid support
>>>>> subscriptions. Contact us at https://www.isc.org/contact/ for more
>>>>> information.
>>>>> >
>>>>> >
>>>>> > bind-users mailing list
>>>>> > bind-users at lists.isc.org
>>>>> > https://lists.isc.org/mailman/listinfo/bind-users
>>>>>
>>>>> --
>>> Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe
>>> from this list
>>>
>>> ISC funds the development of this software with paid support
>>> subscriptions. Contact us at https://www.isc.org/contact/ for more
>>> information.
>>>
>>>
>>> bind-users mailing list
>>> bind-users at lists.isc.org
>>> https://lists.isc.org/mailman/listinfo/bind-users
>>>
>>
>>
>> --
>>
>> *Regards,Nagaraju A.*
>>
>> --
>> Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe
>> from this list
>>
>> ISC funds the development of this software with paid support
>> subscriptions. Contact us at https://www.isc.org/contact/ for more
>> information.
>>
>>
>> bind-users mailing list
>> bind-users at lists.isc.org
>> https://lists.isc.org/mailman/listinfo/bind-users
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/bind-users/attachments/20220711/7212cb6e/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 55947 bytes
Desc: not available
URL: <https://lists.isc.org/pipermail/bind-users/attachments/20220711/7212cb6e/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 50781 bytes
Desc: not available
URL: <https://lists.isc.org/pipermail/bind-users/attachments/20220711/7212cb6e/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 9224 bytes
Desc: not available
URL: <https://lists.isc.org/pipermail/bind-users/attachments/20220711/7212cb6e/attachment-0005.png>

More information about the bind-users mailing list