<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:36:32 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-34398] Mongo WiredTiger Memory Spike And OOM Isssue</title>
                <link>https://jira.mongodb.org/browse/SERVER-34398</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;h3&gt;&lt;a name=&quot;OverthelastcoupleofmonthswehaveseenthehotsecondaryononeofourreplicasetshaveamemoryspikeandthengetkilledbythelinuxOOMKILLER&quot;&gt;&lt;/a&gt;Over the last couple of months we have seen the hot secondary on one of our replica sets have a memory spike and then get killed by the linux OOM_KILLER&lt;/h3&gt;

&lt;p&gt;*Setup: *&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Mongod Version: 3.4.4&lt;/li&gt;
	&lt;li&gt;Replica Set Config&lt;/li&gt;
	&lt;li&gt;3 boxes:&lt;/li&gt;
	&lt;li&gt;One primary and one hot secondary running with 8 core and ~64 GBs of memory with swap disabled&lt;/li&gt;
	&lt;li&gt;Each box runs in Amazon EC2&lt;/li&gt;
	&lt;li&gt;Each box has a 1500 GB file system mounted for its data directory&lt;/li&gt;
	&lt;li&gt;Each box has a 50 GB file system mounted for the db logs&lt;/li&gt;
	&lt;li&gt;One backup secondary with 4 cores and 32GBs of memory&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;The load on hot secondary is the load to keep in sync with the primary + some queries on 2 collections that exist in this replica set. These queries sometimes include table scans.&lt;/p&gt;

&lt;p&gt;Every once in a while our hot secondary&apos;s memory usage will spike causing the oom_killer to kill the running mongod process. (see the mongo memory spike screenshot attached)&lt;/p&gt;

&lt;p&gt;There are some other strange things going on with the mongo process running out of memory. One is that the WiredTiger cache size does not seem to increase during this time period. (See the cache usage image attached)&lt;/p&gt;

&lt;p&gt;There also seems to be huge spike in the amount of data read from disk right before the mongod memory usage spikes - about 40GBs worth. (see the attached disk read spike). A weird thing here is that those disk reads are on the mounted file system which holds the mongo logs (and is located at `/`) and not the one that holds the mongo data.&lt;/p&gt;

&lt;p&gt;I have noticed this issue: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-27909&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-27909&lt;/a&gt; which seems like it could be related? &lt;/p&gt;

&lt;p&gt;I have also attached the diagnostic.data logs from during and after the incident below. I believe that the incident should be towards the end of metrics.2018-04-08T00-12-09Z-00000.  Let me know if there is any other data you need me to provide. Any help would be greatly appreciated. &lt;/p&gt;</description>
                <environment></environment>
        <key id="524369">SERVER-34398</key>
            <summary>Mongo WiredTiger Memory Spike And OOM Isssue</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="4">Incomplete</resolution>
                                        <assignee username="kelsey.schubert@mongodb.com">Kelsey Schubert</assignee>
                                    <reporter username="paetling@gmail.com">Alex Etling</reporter>
                        <labels>
                    </labels>
                <created>Mon, 9 Apr 2018 20:46:07 +0000</created>
                <updated>Mon, 23 Jul 2018 09:29:12 +0000</updated>
                            <resolved>Thu, 21 Jun 2018 00:44:12 +0000</resolved>
                                    <version>3.4.4</version>
                                                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>5</watches>
                                                                                                                <comments>
                            <comment id="1926955" author="thomas.schubert" created="Thu, 21 Jun 2018 00:44:12 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=paetling%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;paetling@gmail.com&quot;&gt;paetling@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;We haven&#8217;t heard back from you for some time, so I&#8217;m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information and we will reopen the ticket.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Kelsey&lt;/p&gt;</comment>
                            <comment id="1860770" author="thomas.schubert" created="Tue, 10 Apr 2018 20:36:24 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=paetling%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;paetling@gmail.com&quot;&gt;paetling@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;I&apos;ve reviewed the metrics files you&apos;ve provided, but unfortunately do not have enough information to conclusively diagnose this issue.&lt;/p&gt;

&lt;p&gt;So we can continue to investigate, would you please restart mongod with &lt;tt&gt;--setParameter heapProfilingEnabled=true&lt;/tt&gt;. After encountering the issue again, please upload the following information:&lt;/p&gt;

&lt;ol&gt;
	&lt;li&gt;the complete mongod log files since the restart. (You can look for heap profiler messages in the log file to verify that it is running correctly.)&lt;/li&gt;
	&lt;li&gt;diagnostic.data covering the entire time since the restart&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;These files will record information that should enable us to track the source of the memory increase. For this purpose it is important that we have complete logs and diagnostic.data covering the time since the restart. Since the required files may be too large to attach to this ticket, I&apos;ve generated a secure &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/aab00fdb-0401-4f52-a81d-e88a04e3cf40.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;upload portal&lt;/a&gt; for you to use.&lt;/p&gt;

&lt;p&gt;Thank you for your help,&lt;br/&gt;
Kelsey&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="183742" name="disk_reads.png" size="93010" author="paetling@gmail.com" created="Mon, 9 Apr 2018 20:37:57 +0000"/>
                            <attachment id="183741" name="metrics.2018-04-08T00-12-09Z-00000" size="3610332" author="paetling@gmail.com" created="Mon, 9 Apr 2018 20:41:28 +0000"/>
                            <attachment id="183740" name="metrics.2018-04-08T04-01-51Z-00000" size="10397358" author="paetling@gmail.com" created="Mon, 9 Apr 2018 20:41:39 +0000"/>
                            <attachment id="183744" name="mongo_memory_spike.png" size="37150" author="paetling@gmail.com" created="Mon, 9 Apr 2018 20:25:34 +0000"/>
                            <attachment id="183743" name="wired_tiger_bytes_in_the_cache.png" size="55708" author="paetling@gmail.com" created="Mon, 9 Apr 2018 20:28:27 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 10 Apr 2018 20:36:24 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 34 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>backlog-server-pm</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 34 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>paetling@gmail.com</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htus33:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|htlzc7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[kelsey.schubert@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htuebb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>