<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:34:38 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-33809] Secondary member crashed due to OOM in production</title>
                <link>https://jira.mongodb.org/browse/SERVER-33809</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hi All,&lt;br/&gt;
we have a mongo ReplicaSet configured with 6 members - all in the same subnet. our application configured to prefer reads from secondaries.&lt;br/&gt;
yesterday one of the nodes suffered from a lack of memory and the kernel decided to perform an OOM event which killed the mongo.&lt;/p&gt;

&lt;p&gt;the mongo is the only thing we run in the machine (using docker). i&apos;v checked the system logs and found this:&lt;br/&gt;
Mar 11 22:40:02 ip-xxx-xx-x-xxx kernel: &lt;span class=&quot;error&quot;&gt;&amp;#91;10313490.393104&amp;#93;&lt;/span&gt; conn14048276 invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0&lt;br/&gt;
Mar 11 22:40:02 ip-xxx-xx-x-xxx kernel: &lt;span class=&quot;error&quot;&gt;&amp;#91;10313490.393109&amp;#93;&lt;/span&gt; conn14048276 cpuset=aa6b9fb618f6296d15a964eea9cab273f3d9476fc66e3b24d2ca4ec8e2784e73 mems_allowed=0&lt;br/&gt;
Mar 11 22:40:02 ip-xxx-xx-x-xxx kernel: &lt;span class=&quot;error&quot;&gt;&amp;#91;10313490.393112&amp;#93;&lt;/span&gt; CPU: 0 PID: 9200 Comm: conn14048276 Not tainted 3.13.0-88-generic #135-Ubuntu&lt;br/&gt;
Mar 11 22:40:02 ip-xxx-xx-x-xxx kernel: &lt;span class=&quot;error&quot;&gt;&amp;#91;10313490.393113&amp;#93;&lt;/span&gt; Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006&lt;br/&gt;
........&lt;br/&gt;
Mar 11 22:40:02 ip-xxx-xx-x-xxx kernel: &lt;span class=&quot;error&quot;&gt;&amp;#91;10313490.393323&amp;#93;&lt;/span&gt; Out of memory: Kill process 4340 (mongod) score 912 or sacrifice child&lt;br/&gt;
Mar 11 22:40:02 ip-xxx-xx-x-xxx kernel: &lt;span class=&quot;error&quot;&gt;&amp;#91;10313490.404537&amp;#93;&lt;/span&gt; Killed process 4340 (mongod) total-vm:7052416kB, anon-rss:3681108kB, file-rss:0kB&lt;/p&gt;

&lt;p&gt;i do have the diagnostic data but not sure how to analyse this (see attached). we don&apos;t have any swap memory in the machine, is it recommended to add?&lt;/p&gt;

&lt;p&gt;its a production machine and first time we encounter a failure so its important to us to understand the root cause and what is necessary to avoid such cases&lt;/p&gt;

&lt;p&gt;Thanks&lt;/p&gt;</description>
                <environment></environment>
        <key id="509604">SERVER-33809</key>
            <summary>Secondary member crashed due to OOM in production</summary>
                <type id="6" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14720&amp;avatarType=issuetype">Question</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13202">Works as Designed</resolution>
                                        <assignee username="dmitry.agranat@mongodb.com">Dmitry Agranat</assignee>
                                    <reporter username="roiey">Roie Yossef</reporter>
                        <labels>
                    </labels>
                <created>Mon, 12 Mar 2018 09:00:17 +0000</created>
                <updated>Fri, 27 Oct 2023 13:53:55 +0000</updated>
                            <resolved>Tue, 13 Mar 2018 17:15:53 +0000</resolved>
                                                                                        <votes>0</votes>
                                    <watches>6</watches>
                                                                                                                <comments>
                            <comment id="1832381" author="dmitry.agranat" created="Tue, 13 Mar 2018 17:14:24 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=roiey&quot; class=&quot;user-hover&quot; rel=&quot;roiey&quot;&gt;roiey&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;All good questions. Please note that the SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the &lt;a href=&quot;http://groups.google.com/group/mongodb-user&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;mongodb-user group&lt;/a&gt; or &lt;a href=&quot;http://stackoverflow.com/questions/tagged/mongodb&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Stack Overflow with the &lt;tt&gt;mongodb&lt;/tt&gt; tag&lt;/a&gt;. Questions like this involving more discussion would be best posted on the mongodb-users group.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                            <comment id="1832183" author="roiey" created="Tue, 13 Mar 2018 15:37:01 +0000"  >&lt;p&gt;Hi Dima,&lt;br/&gt;
Thanks for you answer. we running a replicaSet with 5 members. its the first time we encountered OOM in production. why this specific node failed and the other sastains healthy? is it  possible that one heavy specific query handled by the machine and cause that? if yes , how can we recognize the problematic query? if not , how can i avoid the node from failing? swap will help or just slow down the mongo?&lt;/p&gt;

&lt;p&gt;regarding to the disks , what do you consider as poor IO performance? what disks should we have? we use AWS EBS volume , io1 type with 5000 IOPS.  &lt;/p&gt;</comment>
                            <comment id="1831750" author="dmitry.agranat" created="Tue, 13 Mar 2018 10:24:57 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=roiey&quot; class=&quot;user-hover&quot; rel=&quot;roiey&quot;&gt;roiey&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;After looking at the provided data, I do not see an indication of a bug. I believe that the reported issue is due to insufficient available memory (4GB) relative to your workload.&lt;/p&gt;

&lt;p&gt;As the memory required for operations grows, the working set in the WT cache cannot be pushed back onto disk.  Instead, the operating system begins swapping application memory to disk until it runs out of space and kills the process that is using the most memory, which, in this case, is MongoDB.  &lt;/p&gt;

&lt;p&gt;Based on the additional metrics observed (see below), it appears that this server is also suffering from poor IO performance and mostly reading the data from either the FS cache or disk.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/181563/181563_SERVER-33809.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;All the above indicate that the server is underprovisioned to sustain your workload.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                                        </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="181563" name="SERVER-33809.png" size="257971" author="dmitry.agranat@mongodb.com" created="Tue, 13 Mar 2018 10:23:04 +0000"/>
                            <attachment id="181457" name="metrics.2018-03-11T03-33-58Z-00000.gz" size="10414227" author="roiey" created="Mon, 12 Mar 2018 08:59:22 +0000"/>
                            <attachment id="181458" name="metrics.2018-03-11T14-13-58Z-00000.gz" size="8022989" author="roiey" created="Mon, 12 Mar 2018 08:59:19 +0000"/>
                            <attachment id="181459" name="metrics.2018-03-11T22-40-51Z-00000.gz" size="10433798" author="roiey" created="Mon, 12 Mar 2018 08:59:31 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 13 Mar 2018 10:24:57 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 48 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 48 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>dmitry.agranat@mongodb.com</customfieldvalue>
            <customfieldvalue>roiey</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htsdcv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|htjqan:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[kelsey.schubert@mongodb.com]]></customfieldvalue>
        <customfieldvalue><![CDATA[dmitry.agranat@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htrzjr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>