<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:53:03 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-20068] Mongodb 3.0.5 with wiredTiger causing Out of memory Issues </title>
                <link>https://jira.mongodb.org/browse/SERVER-20068</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We have upgraded our replicated sharded mongo db setup to the latest 3.0.5 i hope of fixing OOM issues we are having after we migrated our Storage Engine from MMAP to WT but our memory usage issues didn&apos;t go away. Memory usage increases over time and only restart releases allocated memory.&lt;/p&gt;

&lt;p&gt;We are running 4 shards on Ubuntu Server(primary instance in our 3 member replica set) having 60 GB System Memory with WT as Storage Engine. We set Cache Size to 13 GB for each Shard Server leaving 8 GB memory for System processes and if Mongo requires more memory for Open Cursors, Open Sessions etc. But it uses way more and System kills the process.&lt;/p&gt;

&lt;p&gt;Two of the four shard servers running on our primary instance failed with OOM error due to system kill.&lt;/p&gt;

&lt;p&gt;Please find attached db.serverStatus(&lt;/p&gt;
{tcmalloc:true}
&lt;p&gt;) captured for all the four shard servers running on Primary captured from 1 hour before the failure occurred. Also attached syslog which logged the system kill actions of the two shard servers.&lt;/p&gt;</description>
                <environment></environment>
        <key id="226480">SERVER-20068</key>
            <summary>Mongodb 3.0.5 with wiredTiger causing Out of memory Issues </summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="ramon.fernandez@mongodb.com">Ramon Fernandez Marina</assignee>
                                    <reporter username="praveenak">Praveen Akinapally</reporter>
                        <labels>
                    </labels>
                <created>Thu, 20 Aug 2015 18:52:04 +0000</created>
                <updated>Tue, 25 Aug 2015 19:33:26 +0000</updated>
                            <resolved>Tue, 25 Aug 2015 19:33:26 +0000</resolved>
                                    <version>3.0.5</version>
                                                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>2</watches>
                                                                                                                <comments>
                            <comment id="1014269" author="ramon.fernandez" created="Tue, 25 Aug 2015 19:33:18 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=praveenak&quot; class=&quot;user-hover&quot; rel=&quot;praveenak&quot;&gt;praveenak&lt;/a&gt;, I see a large number of open cursors, which may or may not be contributing to the problem. In MongoDB 3.0.6 we added a palliative fix for this issue in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-19987&quot; title=&quot;Limit the size of the per-session cursor cache&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-19987&quot;&gt;&lt;del&gt;SERVER-19987&lt;/del&gt;&lt;/a&gt;, so if these cursors are a problem and are coming from a small set of sessions then upgrading to 3.0.6 should help.&lt;/p&gt;

&lt;p&gt;That being said, I think the main cause for this particular problem was the machine&apos;s memory was being overtaxed by running too many servers in it, so I&apos;m going to close this issue for now. If running on a more sensible configuration and upgrading to 3.0.6 do not help with this issue please feel free to open a new ticket and we&apos;ll take it from there.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="1012153" author="praveenak" created="Fri, 21 Aug 2015 20:21:24 +0000"  >&lt;p&gt;Thanks for looking into it. I will decrease the cache size and see. Interested to know what are your findings from the logs.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Praveen&lt;/p&gt;</comment>
                            <comment id="1011155" author="ramon.fernandez" created="Thu, 20 Aug 2015 20:18:59 +0000"  >&lt;p&gt;Thanks for your report &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=praveenak&quot; class=&quot;user-hover&quot; rel=&quot;praveenak&quot;&gt;praveenak&lt;/a&gt;. I&apos;m afraid there&apos;s a lot of confusion over this topic, so I&apos;ll try to clarify things: &lt;b&gt;setting the WiredTiger cache size to 13GB doesn&apos;t mean that &lt;tt&gt;mongod&lt;/tt&gt; will only use 13GB&lt;/b&gt; &amp;#8211; it means that WiredTiger should not use more than 13GB for its cache, but &lt;tt&gt;mongod&lt;/tt&gt; will use memory for other things (connections, session cache, cursor cache...).&lt;/p&gt;

&lt;p&gt;If I understand your setup correctly, you&apos;re running a 4-shard system on one box. If that&apos;s correct, up to 52GB may be used for the WiredTiger cache, thus leaving only 8GB to run four &lt;tt&gt;mongod&lt;/tt&gt; data-bearing processes, config servers, &lt;tt&gt;mongos&lt;/tt&gt; processes, the operating system and whatever other applications may be running. Without proper configuration (e.g.: per-process memory limitations, large amounts of swap available) the system is bound to run out of memory and kill whichever process is more memory hungry &amp;#8211; most likely a &lt;tt&gt;mongod&lt;/tt&gt;.&lt;/p&gt;

&lt;p&gt;That being said, we&apos;ll take a look at the data you collected to make sure there are no other memory issues lurking around &amp;#8211; thanks for collecting this data in advance.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="86923" name="ss-s00.log" size="4710448" author="praveenak" created="Thu, 20 Aug 2015 18:52:04 +0000"/>
                            <attachment id="86924" name="ss-s01.log" size="6743830" author="praveenak" created="Thu, 20 Aug 2015 18:52:04 +0000"/>
                            <attachment id="86921" name="ss-s10.log" size="6821315" author="praveenak" created="Thu, 20 Aug 2015 18:52:04 +0000"/>
                            <attachment id="86922" name="ss-s11.log" size="4599303" author="praveenak" created="Thu, 20 Aug 2015 18:52:04 +0000"/>
                            <attachment id="86920" name="syslog" size="31445" author="praveenak" created="Thu, 20 Aug 2015 18:52:04 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 20 Aug 2015 20:18:59 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        8 years, 25 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            8 years, 25 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>praveenak</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrkwpr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hscnh3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;Deploy a mongo version 3.0.5 sharded replica set&lt;/p&gt;

&lt;p&gt;Environment -&lt;br/&gt;
Server: Ubuntu 14.04&lt;br/&gt;
Ram: 60 GB&lt;/p&gt;

&lt;p&gt;Run four shard servers(all primary in their replica set) on this machine with 13 GB Cache Size.&lt;/p&gt;

&lt;p&gt;Perform write intensive jobs and over time memory used by shard servers increase and finally lead to System Kill due to out of memory.&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hryyjb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>