<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:00:38 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-42493] Replica set crashes</title>
                <link>https://jira.mongodb.org/browse/SERVER-42493</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hey guys, we observed the following weird behaviour with the following setup:&lt;/p&gt;

&lt;p&gt;All times are UTC&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;3-member replica set
	&lt;ul&gt;
		&lt;li&gt;two bigger instances for failover - &lt;b&gt;rs1-1&lt;/b&gt; and &lt;b&gt;rs1-2&lt;/b&gt;&lt;/li&gt;
		&lt;li&gt;one smaller instance for backups&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
&lt;/ul&gt;


&lt;ol&gt;
	&lt;li&gt;Around &lt;b&gt;00:31&lt;/b&gt; the primary &lt;b&gt;rs1-1&lt;/b&gt; had a major spike in memory usage.
	&lt;ul&gt;
		&lt;li&gt;this is inferred from &quot;Cannot allocate memory&quot; messages in the syslog of the instance&lt;/li&gt;
		&lt;li&gt;based on the mongo logs: there are no heavy running queries at the time&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
	&lt;li&gt;After becoming irresponsive&#160;&lt;b&gt;rs1-2&lt;/b&gt; became the new primary and had a similar memory usage spike around &lt;b&gt;00:37&lt;/b&gt;
	&lt;ul&gt;
		&lt;li&gt;again inferred from the syslog&lt;/li&gt;
		&lt;li&gt;again no big queries can be seen in the mongo log&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
	&lt;li&gt;Both instances were irresponsive (not able to SSH, not reporting metrics) for a few hours until restarting them a few hours later&lt;/li&gt;
	&lt;li&gt;Upon restart&#160;&lt;b&gt;rs1-1&lt;/b&gt; crashed one more time around &lt;b&gt;06:44&lt;/b&gt;&lt;/li&gt;
	&lt;li&gt;**After the second crash I scaled up the machines and they have been running OK since then&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;You can see attached:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;mongo logs from both servers&lt;/li&gt;
	&lt;li&gt;diagnostics.data from both servers&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Let me know if you need any more information.&lt;/p&gt;</description>
                <environment></environment>
        <key id="879690">SERVER-42493</key>
            <summary>Replica set crashes</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13202">Works as Designed</resolution>
                                        <assignee username="dmitry.agranat@mongodb.com">Dmitry Agranat</assignee>
                                    <reporter username="adamof">Stefan Adamov</reporter>
                        <labels>
                    </labels>
                <created>Tue, 30 Jul 2019 10:33:20 +0000</created>
                <updated>Fri, 27 Oct 2023 13:53:08 +0000</updated>
                            <resolved>Wed, 31 Jul 2019 11:38:59 +0000</resolved>
                                    <version>3.4.16</version>
                                                    <component>Replication</component>
                    <component>Stability</component>
                                        <votes>0</votes>
                                    <watches>3</watches>
                                                                                                                <comments>
                            <comment id="2352032" author="dmitry.agranat" created="Wed, 31 Jul 2019 11:37:49 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=adamof&quot; class=&quot;user-hover&quot; rel=&quot;adamof&quot;&gt;adamof&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;After reviewing the provided data, I was not able to find any bottleneck on the MongoDB side. The fact that after scaling up the servers from 7.4GB RAM to 15GB RAM the issue does not reoccur, indicates that it might have been simply an issue with the amount of memory used by the queries while running. I would also recommend reviewing the queries for tuning as I&apos;ve noticed the workload sometimes scans ~20 million documents just to return ~700.&lt;/p&gt;

&lt;p&gt;As I was not able to find an issue on the MongoDB side, I will go ahead and close this ticket.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="225586" name="rs1-1-diagnostics.tar" size="49674240" author="adamof" created="Tue, 30 Jul 2019 10:18:38 +0000"/>
                            <attachment id="225585" name="rs1-1-mongo-log" size="1888805" author="adamof" created="Tue, 30 Jul 2019 10:04:58 +0000"/>
                            <attachment id="225584" name="rs1-2-diagnostics.tar" size="35768320" author="adamof" created="Tue, 30 Jul 2019 10:13:06 +0000"/>
                            <attachment id="225583" name="rs1-2-mongo-log" size="750646" author="adamof" created="Tue, 30 Jul 2019 10:04:22 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 31 Jul 2019 11:37:49 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 28 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 28 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>dmitry.agranat@mongodb.com</customfieldvalue>
            <customfieldvalue>adamof</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hvi7iv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hv77dz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[dmitry.agranat@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hvhts7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>