<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:40:43 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-56996] mogodb consumes up to 98% of the available memory</title>
                <link>https://jira.mongodb.org/browse/SERVER-56996</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hello,&lt;/p&gt;

&lt;p&gt;Using mongo in kubernates environ ment with centos as base.&lt;br/&gt;
configured 3 replicas with memory limit to 10G&lt;/p&gt;

&lt;p&gt;Limits:&lt;br/&gt;
 cpu: 5&lt;br/&gt;
 memory: 10Gi&lt;br/&gt;
 Requests:&lt;br/&gt;
 cpu: 1&lt;br/&gt;
 memory: 2Gi&lt;/p&gt;

&lt;p&gt;When the pod started can see it occupies upto 4G of memory and gradually when the parallel connections and queries increases,&lt;br/&gt;
it occupying around 98% of total memory and not releasing the memory post use.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;root@node00 cloud-user&amp;#93;&lt;/span&gt;# kubectl top pod -n test | grep mngo&lt;br/&gt;
e-cmngo-replica-0 92m 9433Mi&lt;br/&gt;
e-mngo-replica-1 89m 6065Mi&lt;br/&gt;
e-mngo-replica-2 96m 6482Mi&lt;/p&gt;

&lt;p&gt;The wiredtiger cache set to default(around 4G )&lt;br/&gt;
is there a way we can release the memory post usage ?&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;WitreTiger info:&lt;br/&gt;
e:PRIMARY&amp;gt; db.serverStatus().wiredTiger.cache&lt;br/&gt;
{&lt;br/&gt;
 &quot;application threads page read from disk to cache count&quot; : 1599135,&lt;br/&gt;
 &quot;application threads page read from disk to cache time (usecs)&quot; : 1619303577,&lt;br/&gt;
 &quot;application threads page write from cache to disk count&quot; : 1796,&lt;br/&gt;
 &quot;application threads page write from cache to disk time (usecs)&quot; : 564252,&lt;br/&gt;
 &quot;bytes belonging to page images in the cache&quot; : 3797310198,&lt;br/&gt;
 &quot;bytes belonging to the cache overflow table in the cache&quot; : 182,&lt;br/&gt;
 &quot;bytes currently in the cache&quot; : 3809181895,&lt;br/&gt;
 &quot;bytes dirty in the cache cumulative&quot; : 88830916,&lt;br/&gt;
 &quot;bytes not belonging to page images in the cache&quot; : 11871696,&lt;br/&gt;
 &quot;bytes read into cache&quot; : 463809362069,&lt;br/&gt;
 &quot;bytes written from cache&quot; : 92928011,&lt;/p&gt;</description>
                <environment></environment>
        <key id="1718641">SERVER-56996</key>
            <summary>mogodb consumes up to 98% of the available memory</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13202">Works as Designed</resolution>
                                        <assignee username="dmitry.agranat@mongodb.com">Dmitry Agranat</assignee>
                                    <reporter username="ece.sagar@gmail.com">S P</reporter>
                        <labels>
                    </labels>
                <created>Mon, 17 May 2021 11:57:02 +0000</created>
                <updated>Fri, 27 Oct 2023 13:52:23 +0000</updated>
                            <resolved>Thu, 20 May 2021 08:58:14 +0000</resolved>
                                    <version>4.2.2</version>
                                                                        <votes>0</votes>
                                    <watches>3</watches>
                                                                                                                <comments>
                            <comment id="3802689" author="JIRAUSER1260032" created="Fri, 21 May 2021 09:50:57 +0000"  >&lt;p&gt;Below image in my local set up clearly shows the virtual memory increased from 6GB to 8GB during insert operation adn after that&#160; remained high at 8GB despite no active connections or any curd operation.&lt;/p&gt;

&lt;p&gt;This memory came down when a restart of mongo triggered. Why the behavior is like this ?&lt;/p&gt;

&lt;p&gt;is this not expected that mongo should release the memory if there are no operations ?&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/316200/316200_image-2021-05-21-15-19-28-106.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;</comment>
                            <comment id="3802507" author="JIRAUSER1260032" created="Fri, 21 May 2021 09:25:34 +0000"  >&lt;p&gt;Hello Dima,&lt;/p&gt;

&lt;p&gt;still not convinced as I can see the number of active connections are only 2 which include the current terminal. with just 2 active connections the system consuming lesser CPU and higher memory(all available) ! Is this a expected behavior. What is the need of consuming all memory with minimal operation?&lt;/p&gt;

&lt;p&gt;As you mentioned the crud operation happening continuously can you please let know from which IP the traffic is more , so that I can check locally. Is there a way we can check ? where it shows the active traffic from client?.&lt;/p&gt;

&lt;p&gt;Also this behavior is consistent . The metrics shared is just for 2 day but mongodb in this lab never releases memory (always 98%). Would like to understand more.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;PRIMARY&amp;gt; db.serverStatus().connections&lt;/p&gt;
{
 &quot;current&quot; : 151,
 &quot;available&quot; : 838709,
 &quot;totalCreated&quot; : 98058,
 &quot;active&quot; : 2
}</comment>
                            <comment id="3793746" author="dmitry.agranat" created="Thu, 20 May 2021 08:57:11 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ece.sagar%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;ece.sagar@gmail.com&quot;&gt;ece.sagar@gmail.com&lt;/a&gt; for the additional context. Based on what I see in your workload, CRUD operations never really stop. There are some periods of time when CRUD operations significantly decrease for a couple of seconds but this is not expected that the consumed resident memory of a process would be returned to the OS &lt;b&gt;immediately&lt;/b&gt; during these 1-2 seconds of low activity.&lt;/p&gt;

&lt;p&gt;Regarding the comparison between the CPU and memory utilization, this is also not expected that the moment the CPU utilization drops to 50%, the same should happen to memory.&lt;/p&gt;

&lt;p&gt;I&apos;ve noticed that about 3% of the total memory is fragmented and we can try to force it to return to the OS more aggressively but I am not sure this aligns with the expectation of reclaiming all the memory.&lt;/p&gt;

&lt;p&gt;As this works as designed, I will go ahead and close this ticket. The SERVER project is for bugs and feature suggestions for the MongoDB server. If you have further questions about memory, we&apos;d like to encourage you to start by asking our community for help by posting on the &lt;a href=&quot;https://community.mongodb.com&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;MongoDB Developer Community Forums&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                            <comment id="3792148" author="JIRAUSER1260032" created="Wed, 19 May 2021 15:48:49 +0000"  >&lt;p&gt;Dima,&lt;/p&gt;

&lt;p&gt;the version is for db servier version&lt;/p&gt;

&lt;p&gt;eden-csf:PRIMARY&amp;gt; db.version()&lt;br/&gt;
 4.2.2-3&lt;br/&gt;
 eden-csf:PRIMARY&amp;gt;&lt;/p&gt;

&lt;p&gt;It says Percona Server for MongoDB shell version v4.2.2-3&lt;/p&gt;

&lt;p&gt;------------------&lt;/p&gt;

&lt;p&gt;Regarding the logs the the time stamp its UTC time zone&lt;/p&gt;

&lt;p&gt;bash-4.4$ date&lt;br/&gt;
 Wed May 19 15:37:15 UTC 2021&lt;/p&gt;

&lt;p&gt;-----------------&lt;/p&gt;

&lt;p&gt;CRUD operations were performed for specific time period . say for 1~1.5 hour .&lt;/p&gt;

&lt;p&gt;In that period the memory usage is high make sense along with cpu utilization is also high.&lt;/p&gt;

&lt;p&gt;But once the operations completes the memory utilization of the pod should also reduce , like the cpu usages fall back to lowest.&lt;/p&gt;

&lt;p&gt;for e.g&lt;/p&gt;

&lt;p&gt;Before starting the crud operation for replica-0 , the cpu is 50 and during crud operation it shooots up to 5008. once its complete the cpu comes back to 50~100 .&lt;/p&gt;

&lt;p&gt;But the same behavior is not for the memory usage of the pod .&lt;/p&gt;

&lt;p&gt;the start memory is 5788 and it increases to 9862.&lt;/p&gt;

&lt;p&gt;but post crud operation its remain same&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/315640/315640_image-2021-05-19-21-16-45-610.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;</comment>
                            <comment id="3791209" author="dmitry.agranat" created="Wed, 19 May 2021 13:14:48 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ece.sagar%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;ece.sagar@gmail.com&quot;&gt;ece.sagar@gmail.com&lt;/a&gt;, I have a couple of clarifying questions:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;What &lt;tt&gt;version:4.2.2-3&lt;/tt&gt; stands for?&lt;/li&gt;
	&lt;li&gt;What timeline we should be looking at for the expected memory release (in UTC timezone)? I am asking this because between the startup on &lt;tt&gt;2021-05-18T13:10:34.680Z&lt;/tt&gt; and the end of the data we have &lt;tt&gt;2021-05-19T07:44:34.283Z&lt;/tt&gt; you query and perform insert/update/delete operations utilizing about ~6 GB of resident memory, so I am not sure why we should expect the memory to be released.&lt;/li&gt;
&lt;/ul&gt;



</comment>
                            <comment id="3790005" author="JIRAUSER1260032" created="Wed, 19 May 2021 08:21:26 +0000"  >&lt;p&gt;metrics.tar uploaded to&#160;&lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/b4d621f1-4cf7-4fa9-a447-b6ca6ad51ec9.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;support uploader&lt;/a&gt;&#160;location&lt;/p&gt;</comment>
                            <comment id="3788952" author="JIRAUSER1260032" created="Wed, 19 May 2021 04:04:58 +0000"  >&lt;p&gt;Hello Dima,&lt;/p&gt;

&lt;p&gt;Will try to get the required logs and upload asap. need some time .&lt;/p&gt;

&lt;p&gt;Meanwhile can you please update is this the expected behavior of mongo ?&lt;/p&gt;

&lt;p&gt;As I read couple of blogs stating out of total available memory wiredtiger consumes as per the configuration and the remaining memory consumed by file system cache, as mentioned in&#160;&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://docs.mongodb.com/manual/core/wiredtiger/#std-label-storage-wiredtiger-journal&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://docs.mongodb.com/manual/core/wiredtiger/#std-label-storage-wiredtiger-journal&lt;/a&gt;&#160;&lt;br/&gt;
Via the filesystem cache, MongoDB automatically uses all free memory that is not used by the WiredTiger cache or by other processes.&lt;br/&gt;
Precisely I would want to release the memory consumed by the pods during high usage and release post usage.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Regards&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="3785777" author="dmitry.agranat" created="Tue, 18 May 2021 19:26:37 +0000"  >
&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ece.sagar%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;ece.sagar@gmail.com&quot;&gt;ece.sagar@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Would you please archive (tar or zip) the mongod.log files covering the reported event and the &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt; directory (the contents are described &lt;a href=&quot;https://docs.mongodb.com/manual/administration/analyzing-mongodb-performance/#full-time-diagnostic-data-capture&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;) and upload them to this &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/b4d621f1-4cf7-4fa9-a447-b6ca6ad51ec9.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;support uploader&lt;/a&gt; location?&lt;/p&gt;

&lt;p&gt;Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time.&lt;/p&gt;

&lt;p&gt;Dima&lt;/p&gt;

</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="1718624">SERVER-56995</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="1718624">SERVER-56995</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="315638" name="image-2021-05-19-21-14-36-006.png" size="10397" author="ece.sagar@gmail.com" created="Wed, 19 May 2021 15:44:37 +0000"/>
                            <attachment id="315640" name="image-2021-05-19-21-16-45-610.png" size="10432" author="ece.sagar@gmail.com" created="Wed, 19 May 2021 15:46:47 +0000"/>
                            <attachment id="316197" name="image-2021-05-21-15-04-00-522.png" size="92926" author="ece.sagar@gmail.com" created="Fri, 21 May 2021 09:34:02 +0000"/>
                            <attachment id="316198" name="image-2021-05-21-15-16-16-407.png" size="132154" author="ece.sagar@gmail.com" created="Fri, 21 May 2021 09:46:19 +0000"/>
                            <attachment id="316200" name="image-2021-05-21-15-19-28-106.png" size="135115" author="ece.sagar@gmail.com" created="Fri, 21 May 2021 09:49:32 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 18 May 2021 19:26:37 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 37 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 37 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>dmitry.agranat@mongodb.com</customfieldvalue>
            <customfieldvalue>ece.sagar@gmail.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hzb97r:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hyvztr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;Start mongo pod&lt;/p&gt;

&lt;p&gt;Increase the no of records to 50K&lt;/p&gt;

&lt;p&gt;run locust test with client requests as below&lt;/p&gt;

&lt;p&gt;number_of_users = 10&lt;br/&gt;
hatch_rate = 3&lt;br/&gt;
duration = 10m&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[dmitry.agranat@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hzavgv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>