<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:55:27 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-40595] memory usage saturation</title>
                <link>https://jira.mongodb.org/browse/SERVER-40595</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We had a production system that has been working fine slow to a crawl and kick users out after the memory usage grew to 99% overall.&#160;&lt;/p&gt;

&lt;p&gt;We are currently on 3.6.10&#160;&lt;/p&gt;

&lt;p&gt;Single server, no replica set&lt;/p&gt;

&lt;p&gt;Config has wiredTiger.cacheSizeGB: 13&lt;/p&gt;

&lt;p&gt;Total system memory is 24gb&lt;/p&gt;

&lt;p&gt;values reported by mongostat were&lt;/p&gt;

&lt;p&gt;vsize&#160; 40gb&lt;/p&gt;

&lt;p&gt;res was 19&lt;/p&gt;

&lt;p&gt;dirty .8%&lt;/p&gt;

&lt;p&gt;used 80%&lt;/p&gt;

&lt;p&gt;I have the metrics files from the diagnostic.data for the time frame, are there tools available yet so I can analyze these?&lt;/p&gt;</description>
                <environment></environment>
        <key id="736608">SERVER-40595</key>
            <summary>memory usage saturation</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="daniel.hatcher@mongodb.com">Danny Hatcher</assignee>
                                    <reporter username="mhughes">Matt Hughes</reporter>
                        <labels>
                    </labels>
                <created>Thu, 11 Apr 2019 19:38:43 +0000</created>
                <updated>Wed, 11 Sep 2019 16:37:30 +0000</updated>
                            <resolved>Wed, 11 Sep 2019 16:37:30 +0000</resolved>
                                    <version>3.6.10</version>
                                                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="2416724" author="daniel.hatcher" created="Wed, 11 Sep 2019 16:37:22 +0000"  >&lt;p&gt;It is possible that there is a bug with the way our Windows memory metrics are reporting but as you&apos;ve said that&apos;s a little beyond the scope of this particular ticket. If you think you can reproduce the incorrect reporting manually then I recommend you open a new ticket where we can dig in there. For now I&apos;ll close this ticket.&lt;/p&gt;</comment>
                            <comment id="2407762" author="mhughes" created="Wed, 4 Sep 2019 19:32:32 +0000"  >&lt;p&gt;The oddity is though that the mongod process is not showing as it is the one using the memory which then clears when mongod is stopped, which I guess at this point is outside of the scope of this jira. We only run mongod as the sole process on a windows server and from what ive seen you cannot limit memory usage per process very well in windows. We havent seen a system completely crash like the original one reported since moving on to 3.6.12 and 13 so i suppose&#160; this can be closed. Its peculiar that the mongod process doesnt report all of the memory being used when looking at Windows task manager or perfmon.&lt;/p&gt;</comment>
                            <comment id="2407742" author="daniel.hatcher" created="Wed, 4 Sep 2019 19:23:15 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mhughes&quot; class=&quot;user-hover&quot; rel=&quot;mhughes&quot;&gt;mhughes&lt;/a&gt; have you had a chance to read my previous comment?&lt;/p&gt;</comment>
                            <comment id="2359679" author="daniel.hatcher" created="Tue, 6 Aug 2019 15:33:06 +0000"  >&lt;p&gt;While MongoDB will utilize the &lt;a href=&quot;https://docs.mongodb.com/manual/reference/configuration-options/#storage.wiredTiger.engineConfig.cacheSizeGB&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;--wiredTigerCacheSizeGB setting&lt;/a&gt; for WiredTiger itself, the server will also utilize the filesystem cache to keep compressed data files in memory as well as for other operations. Thus, it is expected and normal for MongoDB to eventually grow and acquire all free memory on a system. For this reason, virtual memory is not an important aspect in diagnosing MongoDB issues. We recommend running a &lt;tt&gt;mongod&lt;/tt&gt; process as the sole significant process on any given server (or restrain resource usage via OS settings) so that other programs do not interfere with it running.&lt;/p&gt;

&lt;p&gt;That being said, we do not expect resident memory to rise high enough to cause out-of-memory issues except for extreme situations. If you think you are running into memory issues (other than virtual memory rising high for the process), please upload the diagnostics to the &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/8eed1b87-55ea-477a-b250-03b448f97535.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Support Uploader&lt;/a&gt; link you were provided earlier along with specific timestamps and we can try to see if there is a bug present. &lt;/p&gt;</comment>
                            <comment id="2353278" author="mhughes" created="Wed, 31 Jul 2019 23:10:59 +0000"  >&lt;p&gt;Windows task manager, resource monitor, perfmon, and the image above. If we have the wiredtiger cache limited at 13gb why is the virtual memory usage showing so much higher? I have a site I recent added a secondary member to converting to areolica from a standalone it had 1gb free out of 34gb the day after the secondary got in sync and there was only 18gb reported in perfmon with nothing else showing it as used. Is this just normal behaviour then? Just trying to understand why mongod process is only showing a smaller percentage of usage in the os and nothing else has it allocated, stop mongo and the extra memory usage goes away.&lt;/p&gt;</comment>
                            <comment id="2351090" author="daniel.hatcher" created="Tue, 30 Jul 2019 19:18:11 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mhughes&quot; class=&quot;user-hover&quot; rel=&quot;mhughes&quot;&gt;mhughes&lt;/a&gt;, have you had a chance to review &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=eric.sedor&quot; class=&quot;user-hover&quot; rel=&quot;eric.sedor&quot;&gt;eric.sedor&lt;/a&gt;&apos;s latest comment?&lt;/p&gt;</comment>
                            <comment id="2329281" author="eric.sedor" created="Tue, 16 Jul 2019 18:29:25 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mhughes&quot; class=&quot;user-hover&quot; rel=&quot;mhughes&quot;&gt;mhughes&lt;/a&gt;; Can you please clarify exactly how you are measuring free memory?&lt;/p&gt;

&lt;p&gt;For reference, here is memory use as reported in the provided diagnostic data. We are going to look for anything unusual but ideally we will need to see a specific incident to determine if a given amount of memory use is the result of a bug.&lt;/p&gt;

&lt;p&gt; &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/224401/224401_Screen+Shot+2019-07-16+at+11.25.29+AM.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;/p&gt;
</comment>
                            <comment id="2327355" author="mhughes" created="Mon, 15 Jul 2019 17:24:32 +0000"  >&lt;p&gt;We havent had any sites use up all memory lately but I uploaded one from a site with some memory that couldnt be tied to any specific process. They currently seem stable but there is only 1-2gb free and seems to be hanging around there.&lt;/p&gt;</comment>
                            <comment id="2322123" author="eric.sedor" created="Thu, 11 Jul 2019 23:04:57 +0000"  >&lt;p&gt;Hi,&lt;/p&gt;

&lt;p&gt;We still need additional information to diagnose the problem. If this is still an issue for you, would you please provide diagnostic data, logs, and timestamps for additional occurrences of the behavior you are seeing?&lt;/p&gt;

&lt;p&gt;Thank you!&lt;/p&gt;</comment>
                            <comment id="2289054" author="eric.sedor" created="Tue, 18 Jun 2019 21:00:22 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mhughes&quot; class=&quot;user-hover&quot; rel=&quot;mhughes&quot;&gt;mhughes&lt;/a&gt;, even though we can&apos;t clearly point to where a memory limitation is being reached, we do see some large swings in session counts and we recommend patching to 3.6.12 to obtain the fix for &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-39932&quot; title=&quot;SessionRuntimeInfo memory is not being released &quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-39932&quot;&gt;&lt;del&gt;SERVER-39932&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;So far, we have not been able to confirm that MongoDB was running out of memory, as resident memory was only 19GB out of the available 24 GB of RAM. But we do see growing memory use and think it is related to an increase in aggregation operations, which would be responsible for memory use outside of the WiredTiger cache.&lt;/p&gt;

&lt;p&gt;Have you seen other occurrences like this that you could provide diagnostic data for?&lt;/p&gt;</comment>
                            <comment id="2274369" author="eric.sedor" created="Thu, 6 Jun 2019 18:46:22 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mhughes&quot; class=&quot;user-hover&quot; rel=&quot;mhughes&quot;&gt;mhughes&lt;/a&gt;; we&apos;re analyzing these files and will get back to you with our thoughts and any questions we have.&lt;/p&gt;</comment>
                            <comment id="2265494" author="mhughes" created="Thu, 30 May 2019 16:33:59 +0000"  >&lt;p&gt;Files have been uploaded. It looks like I neglected to mention previously that all of our applications had been stopped, they had to be because nothing was responsive, to get memory back the mongo service had to be restarted&lt;/p&gt;</comment>
                            <comment id="2264465" author="eric.sedor" created="Wed, 29 May 2019 21:37:41 +0000"  >&lt;p&gt;Hi,&lt;/p&gt;

&lt;p&gt;We still need additional information to diagnose the problem. If this is still an issue for you, would you please provide the requested diagnostic data?&lt;/p&gt;

&lt;p&gt;If you still have privacy concerns, you may feel more comfortable using this support loader &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/8eed1b87-55ea-477a-b250-03b448f97535.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;endpoint&lt;/a&gt;.&#160;Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Eric&lt;/p&gt;</comment>
                            <comment id="2212020" author="eric.sedor" created="Fri, 12 Apr 2019 21:19:19 +0000"  >&lt;p&gt;The contents of the directory are described &lt;a href=&quot;https://docs.mongodb.com/manual/administration/analyzing-mongodb-performance/#full-time-diagnostic-data-capture&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="2210596" author="mhughes" created="Fri, 12 Apr 2019 01:44:01 +0000"  >&lt;p&gt;Posted here because I couldn&apos;t see any reason why there was so much more memory usage by mongo past what the wiredTiger cache was set to in config and I don&apos;t see any tools available anywhere to analyze the diagnostic files. I can post them but can you confirm they have no data about what&apos;s stored in the database inside of them? Our data contains patient health information and if it contains data I cannot post them publicly.&lt;/p&gt;</comment>
                            <comment id="2210285" author="eric.sedor" created="Thu, 11 Apr 2019 20:28:19 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mhughes&quot; class=&quot;user-hover&quot; rel=&quot;mhughes&quot;&gt;mhughes&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Do you have reason to suspect a bug in the MongoDB server? The SERVER project is for bugs and feature suggestions for the MongoDB server. If you archive (tar or zip) the &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt; directory (described &lt;a href=&quot;https://docs.mongodb.com/manual/administration/analyzing-mongodb-performance/#full-time-diagnostic-data-capture&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;) we can examine it for the purposes of determining if the issue is a bug.&lt;/p&gt;

&lt;p&gt;However, a better place to start given seemingly poor performance or high memory use for a particular workload is to engage the MongoDB community by posting on the &lt;a href=&quot;https://groups.google.com/group/mongodb-user&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;mongodb-user group&lt;/a&gt; or on &lt;a href=&quot;https://stackoverflow.com/questions/tagged/mongodb&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Stack Overflow with the &lt;tt&gt;mongodb&lt;/tt&gt; tag&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Does this make sense?&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="219299" name="Screen Shot 2019-06-06 at 11.38.22 AM.png" size="154419" author="eric.sedor@mongodb.com" created="Thu, 6 Jun 2019 18:39:26 +0000"/>
                            <attachment id="219300" name="Screen Shot 2019-06-06 at 11.41.14 AM.png" size="46053" author="eric.sedor@mongodb.com" created="Thu, 6 Jun 2019 18:41:37 +0000"/>
                            <attachment id="219301" name="Screen Shot 2019-06-06 at 11.42.27 AM.png" size="194487" author="eric.sedor@mongodb.com" created="Thu, 6 Jun 2019 18:42:54 +0000"/>
                            <attachment id="224401" name="Screen Shot 2019-07-16 at 11.25.29 AM.png" size="277990" author="eric.sedor@mongodb.com" created="Tue, 16 Jul 2019 18:29:02 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>16.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 11 Apr 2019 20:28:19 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 22 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>daniel.hatcher@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 22 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>daniel.hatcher@mongodb.com</customfieldvalue>
            <customfieldvalue>eric.sedor@mongodb.com</customfieldvalue>
            <customfieldvalue>mhughes</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|huu6y7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hujofb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[eric.sedor@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hutt7j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>