<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:16:20 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-27818] mongod 3.2.11 out of memory - killed by OOM killer</title>
                <link>https://jira.mongodb.org/browse/SERVER-27818</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Our deployment has 4 shards running version 3.2.11, each consisting of a primary, secondary and arbiter, with the wiredTiger engine. Primaries and secondaries are on m3.xlarge servers with 15 GB RAM and 15 GB of swap space. While running smoothly for several months, recently mongod can suddenly be killed by the kernel due to running out of memory. This usually happens on secondaries, but can also happen on primaries. The servers do not seem to be under unusual query load when it happens.&lt;/p&gt;

&lt;p&gt;We do not have any text indexes so it does not seem related to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-18926&quot; title=&quot;Full text search extremely slow and uses a lot of memory under WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-18926&quot;&gt;&lt;del&gt;SERVER-18926&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Attached is example dmesg output from one of the shards. We can also provide contents of diagnostic.data directory.&lt;/p&gt;</description>
                <environment>EC2 m3.xlarge server with 15GB of RAM and 15GB of swap space.&lt;br/&gt;
</environment>
        <key id="349292">SERVER-27818</key>
            <summary>mongod 3.2.11 out of memory - killed by OOM killer</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="kelsey.schubert@mongodb.com">Kelsey Schubert</assignee>
                                    <reporter username="meni@contactspls.com">Meni Livne</reporter>
                        <labels>
                    </labels>
                <created>Thu, 26 Jan 2017 07:13:58 +0000</created>
                <updated>Tue, 31 Jan 2017 17:12:47 +0000</updated>
                            <resolved>Tue, 31 Jan 2017 17:12:47 +0000</resolved>
                                    <version>3.2.11</version>
                                                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="1488205" author="thomas.schubert" created="Mon, 30 Jan 2017 16:51:21 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=meni%40contactspls.com&quot; class=&quot;user-hover&quot; rel=&quot;meni@contactspls.com&quot;&gt;meni@contactspls.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;It&apos;s looks like the issue you are experiencing is likely caused by the configuration of Xen. Memory ballooning and/or a full swap could explain this behavior.&lt;/p&gt;

&lt;p&gt;From the information you have provided, I do not see anything to indicate a bug in the MongoDB server. For MongoDB-related support discussion please post on the &lt;a href=&quot;http://groups.google.com/group/mongodb-user&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;mongodb-user group&lt;/a&gt; or &lt;a href=&quot;http://stackoverflow.com/questions/tagged/mongodb&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Stack Overflow with the &lt;tt&gt;mongodb&lt;/tt&gt; tag&lt;/a&gt;. A question like this involving more discussion would be best posted on the mongodb-users group.&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1487458" author="meni@contactspls.com" created="Sat, 28 Jan 2017 07:56:47 +0000"  >&lt;p&gt;There are no containers. As for limits, max resident set is unlimited. These are the full limits for the mongod process.&lt;/p&gt;

&lt;p&gt;&lt;tt&gt;Limit                     Soft Limit           Hard Limit           Units&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;Max cpu time              unlimited            unlimited            seconds&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;Max file size             unlimited            unlimited            bytes&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;Max data size             unlimited            unlimited            bytes&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;Max stack size            8388608              unlimited            bytes&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;Max core file size        0                    unlimited            bytes&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;Max resident set          unlimited            unlimited            bytes&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;Max processes             64000                64000                processes&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;Max open files            64000                64000                files&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;Max locked memory         65536                65536                bytes&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;Max address space         unlimited            unlimited            bytes&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;Max file locks            unlimited            unlimited            locks&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;Max pending signals       60081                60081                signals&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;Max msgqueue size         819200               819200               bytes&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;Max nice priority         0                    0&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;Max realtime priority     0                    0&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;Max realtime timeout      unlimited            unlimited            us&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;(Kernel version: Linux 4.4.0-59-generic #80-Ubuntu SMP Fri Jan 6 17:47:47 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux)&lt;/p&gt;</comment>
                            <comment id="1487364" author="thomas.schubert" created="Fri, 27 Jan 2017 22:32:35 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=meni%40contactspls.com&quot; class=&quot;user-hover&quot; rel=&quot;meni@contactspls.com&quot;&gt;meni@contactspls.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for uploading the files. I&apos;ve examined the system logs and see that total memory utilization when the system killed mongod was 8259476kB or about 8GB. Are you aware of any containers or ulimits that are constraining the memory available?&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1485682" author="meni@contactspls.com" created="Thu, 26 Jan 2017 08:32:49 +0000"  >&lt;p&gt;We&apos;ve uploaded the diagnostic.data files for the 3 shards whose secondaries experienced the OOM. The files were copied when the mongod processes were down after being killed by the kernel.&lt;/p&gt;

&lt;p&gt;We&apos;ve also uploaded the log files for those servers, from the time they were brought up until being killed and afterwards. &lt;br/&gt;
operationProfiling.slowOpThresholdMs is set to 200ms, so the log is full of these long queries.&lt;br/&gt;
Note that the shard2 server was killed a second time due to OOM a few minutes after restarting it, while it was still syncing the missed ops from the primary.&lt;/p&gt;</comment>
                            <comment id="1485671" author="thomas.schubert" created="Thu, 26 Jan 2017 07:19:57 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=meni%40contactspls.com&quot; class=&quot;user-hover&quot; rel=&quot;meni@contactspls.com&quot;&gt;meni@contactspls.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Would you please upload the &lt;tt&gt;diagnostic.data&lt;/tt&gt; and complete logs for the affected node to this &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/c4dd545c-73de-4382-9bcb-820df5ca189b.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;secure portal&lt;/a&gt;?&lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="148299" name="shard0-dmesg.txt" size="7567" author="meni@contactspls.com" created="Thu, 26 Jan 2017 07:13:16 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 26 Jan 2017 07:19:57 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        7 years, 2 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            7 years, 2 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
            <customfieldvalue>meni@contactspls.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht1lmv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hsu9nz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[kelsey.schubert@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hs4hxz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>