<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 06:54:11 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-84145] Mongodb 5.0.20 process is getting crashed due to higher OS cache memory utilization. </title>
                <link>https://jira.mongodb.org/browse/SERVER-84145</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Mongodb process is getting crashed after upgrading to 5.0.20 from 4.4.18. OS cache memory utlization is going higher with in few hours and mongdb process is getting crashed.&#160;&lt;/p&gt;

&lt;p&gt;Setup details:&lt;br/&gt;
Replica sets are having 7 members, 1 primary, 3 secondaries and 3 arbiters. Priamry and Secondary members are distributed across different sites like A,B,C and D. Arbiter members are distributed across A,B,C,D sites. Each site will be having DB VMs and each VM contains mongodb containers. One VM having 1 primary , 3 secondaries and 2 arbiters members.&#160;&lt;/p&gt;

&lt;p&gt;Scenario:&lt;br/&gt;
1. Bring down one Site , say Site B and start the traffic. Send the traffic such a way that linearly increment up to X limit. After reaching the limit X, send same traffic upto 48 hours.&#160;&lt;/p&gt;

&lt;p&gt;Crash Info:&lt;/p&gt;
{&quot;t&quot;:\{&quot;$date&quot;:&quot;2023-11-28T15:30:56.755+00:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;E&quot;, &#160;&quot;c&quot;:&quot;STORAGE&quot;, &#160;&quot;id&quot;:22435, &#160; &quot;ctx&quot;:&quot;Checkpointer&quot;,&quot;msg&quot;:&quot;WiredTiger error&quot;,&quot;attr&quot;:{&quot;error&quot;:22,&quot;message&quot;:&quot;&lt;span class=&quot;error&quot;&gt;&amp;#91;1701185456:755161&amp;#93;&lt;/span&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;12787:0x7fd437302700&amp;#93;&lt;/span&gt;, &lt;a href=&quot;file:collection-45-1301129529321625809.wt&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;file:collection-45-1301129529321625809.wt&lt;/a&gt;, WT_SESSION.checkpoint: __wt_block_checkpoint_resolve, 928: collection-45-1301129529321625809.wt: the checkpoint failed, the system must restart: Invalid argument&quot;}}&lt;/p&gt;
{&quot;t&quot;:\{&quot;$date&quot;:&quot;2023-11-28T15:30:56.755+00:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;E&quot;, &#160;&quot;c&quot;:&quot;STORAGE&quot;, &#160;&quot;id&quot;:22435, &#160; &quot;ctx&quot;:&quot;Checkpointer&quot;,&quot;msg&quot;:&quot;WiredTiger error&quot;,&quot;attr&quot;:{&quot;error&quot;:-31804,&quot;message&quot;:&quot;&lt;span class=&quot;error&quot;&gt;&amp;#91;1701185456:755177&amp;#93;&lt;/span&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;12787:0x7fd437302700&amp;#93;&lt;/span&gt;, &lt;a href=&quot;file:collection-45-1301129529321625809.wt&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;file:collection-45-1301129529321625809.wt&lt;/a&gt;, WT_SESSION.checkpoint: __wt_block_checkpoint_resolve, 928: the process must exit and restart: WT_PANIC: WiredTiger library panic&quot;}}&lt;/p&gt;
{&quot;t&quot;:\{&quot;$date&quot;:&quot;2023-11-28T15:30:56.755+00:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;F&quot;, &#160;&quot;c&quot;:&quot;-&quot;, &#160; &#160; &#160; &#160;&quot;id&quot;:23089, &#160; &quot;ctx&quot;:&quot;Checkpointer&quot;,&quot;msg&quot;:&quot;Fatal assertion&quot;,&quot;attr&quot;:{&quot;msgid&quot;:50853,&quot;file&quot;:&quot;src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp&quot;,&quot;line&quot;:574}}&lt;/p&gt;
{&quot;t&quot;:\{&quot;$date&quot;:&quot;2023-11-28T15:30:56.755+00:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;F&quot;, &#160;&quot;c&quot;:&quot;-&quot;, &#160; &#160; &#160; &#160;&quot;id&quot;:23090, &#160; &quot;ctx&quot;:&quot;Checkpointer&quot;,&quot;msg&quot;:&quot;\n\n***aborting after fassert() failure\n\n&quot;}&lt;/p&gt;
{&quot;t&quot;:\{&quot;$date&quot;:&quot;2023-11-28T15:30:56.755+00:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;F&quot;, &#160;&quot;c&quot;:&quot;CONTROL&quot;, &#160;&quot;id&quot;:6384300, &quot;ctx&quot;:&quot;Checkpointer&quot;,&quot;msg&quot;:&quot;Writing fatal message&quot;,&quot;attr&quot;:{&quot;message&quot;:&quot;Got signal: 6 (Aborted).\n&quot;}}&lt;/p&gt;


&lt;p&gt;Please note that in Mongodb 4.4.18, we have performed the above scenario and not seen the crash and OS cache memory utilization was constant.&#160;&lt;br/&gt;
In mongodb 4.4.18 this flag --enableMajorityReadConcern false was during the startup.&#160;&lt;br/&gt;
In Mongodb 5.0.20, this flag is --enableMajorityReadConcern false is removed and disabled flowControl feature.&#160;&lt;br/&gt;
In 5.0.20, after traffic is max limit , the cache memory is getting incrementing exponentially and mongodb processes are getting crashed across cluster. Only primary members are getting crashed.&#160;&lt;br/&gt;
If we bring up Site B , then mongodb not crashed and running fine more than 48 hours.&#160;&lt;/p&gt;</description>
                <environment>Ubuntu 20.04 and Mongodb 5.0.20</environment>
        <key id="2516347">SERVER-84145</key>
            <summary>Mongodb 5.0.20 process is getting crashed due to higher OS cache memory utilization. </summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="chris.kelly@mongodb.com">Chris Kelly</assignee>
                                    <reporter username="sreedhar.nalgonda@gmail.com">Sreedhar N</reporter>
                        <labels>
                    </labels>
                <created>Tue, 5 Dec 2023 05:24:31 +0000</created>
                <updated>Thu, 25 Jan 2024 20:52:51 +0000</updated>
                            <resolved>Thu, 25 Jan 2024 20:52:51 +0000</resolved>
                                                                                        <votes>51</votes>
                                    <watches>8</watches>
                                                                                                                <comments>
                            <comment id="6042741" author="JIRAUSER1265262" created="Thu, 25 Jan 2024 20:33:22 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sreedhar.nalgonda%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;sreedhar.nalgonda@gmail.com&quot;&gt;sreedhar.nalgonda@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;It looks like the server is crashing due to &quot;No space left on device&quot; errors immediately preceding the errors you are pointing out. Specifically:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;I do not observe an exponential increase in os cache memory usage, and the diagnostic.data provided only appears to show after the issue occurs.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/506816/506816_image-2024-01-25-14-46-52-089.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;For this issue we&apos;d like to encourage you to start by asking our community for help by posting on the &lt;a href=&quot;https://www.mongodb.com/community/forums&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;MongoDB Developer Community Forums&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;If the discussion there leads you to suspect a bug in the MongoDB server, then we&apos;d want to investigate it as a possible bug here in the SERVER project.&lt;/p&gt;</comment>
                            <comment id="5982038" author="JIRAUSER1258365" created="Wed, 3 Jan 2024 04:47:05 +0000"  >&lt;p&gt;Hi Chris Kelly,&lt;/p&gt;

&lt;p&gt;Thanks for providing an access to upload files. Files are uploaded now. Kindly have a look and provide solution for crash.&#160;&lt;/p&gt;

&lt;p&gt;Thanks,&lt;/p&gt;

&lt;p&gt;Sreedhar&lt;/p&gt;</comment>
                            <comment id="5972253" author="JIRAUSER1265262" created="Wed, 27 Dec 2023 15:57:35 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sreedhar.nalgonda%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;sreedhar.nalgonda@gmail.com&quot;&gt;sreedhar.nalgonda@gmail.com&lt;/a&gt;!&lt;/p&gt;

&lt;p&gt;Thanks for your report, and your patience here. Anecdotally, this an error I&apos;ve seen when we run out of space on a device (&lt;a href=&quot;https://jira.mongodb.org/browse/WT-11906?&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;WT-11906)&lt;/a&gt; but I can&apos;t discern enough&#160; information here.&lt;/p&gt;

&lt;p&gt;To look into this further, would you please archive (tar or zip) the mongod.log files and the &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt; directory (the contents are described &lt;a href=&quot;https://docs.mongodb.com/manual/administration/analyzing-mongodb-performance/#full-time-diagnostic-data-capture&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;) and upload them to this &lt;a href=&quot;https://amphora.corp.mongodb.com/public/upload/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJmb2xkZXJfaWQiOiIyNDE2ODM2NDkyNTkiLCJleHAiOjE3MDYyODQ2MDl9.7f2e5vplJUWRAnF1kvAGe0E8lsXaMrn3nPULSM27Q90&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;support uploader&lt;/a&gt; location?&lt;/p&gt;

&lt;p&gt;Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="506816" name="image-2024-01-25-14-46-52-089.png" size="127320" author="chris.kelly@mongodb.com" created="Thu, 25 Jan 2024 20:46:53 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25137"><![CDATA[Server Triage]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 27 Dec 2023 15:57:35 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        1 week, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>chris.kelly@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            1 week, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>chris.kelly@mongodb.com</customfieldvalue>
            <customfieldvalue>sreedhar.nalgonda@gmail.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i3375j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i2kzb0:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[chris.kelly@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i32tav:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>