<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:50:46 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-60802] Primary node turns to ROLLBACK state permanently</title>
                <link>https://jira.mongodb.org/browse/SERVER-60802</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We have a MongoDB cluster host on-premise on AWS, containing 1 primary node and 2 secondary nodes, on 3 r5 EC2 instances. Due to some heavy workloads, the primary node&apos;s memory utilization reached 100% and then the instance crashed.&lt;/p&gt;

&lt;p&gt;After rebooting the instance, we restart the MongoDB, one of the secondary nodes became the primary as expected. Then the former primary node turned into ROLLBACK state. We noticed the docs on &lt;a href=&quot;https://docs.mongodb.com/manual/core/replica-set-rollbacks/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://docs.mongodb.com/manual/core/replica-set-rollbacks/&lt;/a&gt;&#160;that this is because secondaries can not keep up with the throughput of operations on the former primary. However, it stuck at the state after several rollback files were created under the rollback folder, and after that, we did not notice any new rollback activities on the log.&lt;/p&gt;

&lt;p&gt;In the end, we stopped MongoDB, cleared all data on the node, and started again to sync data from the replica set.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1901816">SERVER-60802</key>
            <summary>Primary node turns to ROLLBACK state permanently</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="4">Incomplete</resolution>
                                        <assignee username="dmitry.agranat@mongodb.com">Dmitry Agranat</assignee>
                                    <reporter username="zijun.tian@tusimple.ai">Zijun Tian</reporter>
                        <labels>
                    </labels>
                <created>Tue, 19 Oct 2021 04:34:07 +0000</created>
                <updated>Fri, 10 Jun 2022 13:07:24 +0000</updated>
                            <resolved>Wed, 17 Nov 2021 11:57:24 +0000</resolved>
                                    <version>4.2.15</version>
                                                                        <votes>0</votes>
                                    <watches>2</watches>
                                                                                                                <comments>
                            <comment id="4166369" author="dmitry.agranat" created="Wed, 3 Nov 2021 11:55:02 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=zijun.tian%40tusimple.ai&quot; class=&quot;user-hover&quot; rel=&quot;zijun.tian@tusimple.ai&quot;&gt;zijun.tian@tusimple.ai&lt;/a&gt;, after looking at the logs, it is not clear why the rollback took so long but there are some hints. For example, there are indications of some networking and storage related issues. If a node is struggling to get some resources from the OS to complete its tasks (in this case, a ROLLBACK), this task might take a long time or be stuck until the requested resource is available.&lt;/p&gt;

&lt;p&gt;Unfortunately, we do not have the &lt;tt&gt;diagnostic.data&lt;/tt&gt; to either prove or refute this assumption but based on the limited information we see in logs, this might be indeed the case.&lt;/p&gt;</comment>
                            <comment id="4157919" author="JIRAUSER1269666" created="Fri, 29 Oct 2021 18:18:02 +0000"  >&lt;p&gt;Hi, I uploaded 3 log files (node1.log, node2.log, node3.log)&lt;/p&gt;

&lt;p&gt;Logs are from 2021-10-18T19:30:00 - 2021-10-19T11:30:00 (UTC)&lt;/p&gt;

&lt;p&gt;Node1 was the secondary node.&lt;/p&gt;

&lt;p&gt;Node2 was the previous primary and went down around 2021-10-18T19:55:00. We tried to recover it from&#160;2021-10-18T19:55:00 to&#160;2021-10-19T01:30:00, after failure, we deleted all data and sync from zero.&lt;/p&gt;

&lt;p&gt;Node3 was the secondary node, and became primary after Node2 went down.&lt;/p&gt;</comment>
                            <comment id="4154130" author="dmitry.agranat" created="Thu, 28 Oct 2021 12:21:32 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=zijun.tian%40tusimple.ai&quot; class=&quot;user-hover&quot; rel=&quot;zijun.tian@tusimple.ai&quot;&gt;zijun.tian@tusimple.ai&lt;/a&gt; This might be more challenging to diagnose this issue w/o the &lt;tt&gt;diagnostic.data&lt;/tt&gt; but we can try. You can upload logs (make sure they cover the time of the event) from all members into &lt;a href=&quot;https://amphora.corp.mongodb.com/public/upload/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJmb2xkZXJfaWQiOiIxNDg5OTE3NzMyODQiLCJleHAiOjE2MzgwMTU1MDF9.UfRwSDiPhTsHqOMlisPLnc-RHjIkZakp6N0kuyXOarM&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this secure portal&lt;/a&gt;. Please make sure to mention the time and the time zone of the event.&lt;/p&gt;</comment>
                            <comment id="4153692" author="JIRAUSER1269666" created="Thu, 28 Oct 2021 07:10:28 +0000"  >&lt;p&gt;Hi, I still have mongod logs but no diagnostic data.&lt;/p&gt;</comment>
                            <comment id="4153690" author="dmitry.agranat" created="Thu, 28 Oct 2021 07:07:42 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=zijun.tian%40tusimple.ai&quot; class=&quot;user-hover&quot; rel=&quot;zijun.tian@tusimple.ai&quot;&gt;zijun.tian@tusimple.ai&lt;/a&gt;, so just to confirm, you no longer have &lt;tt&gt;mongod&lt;/tt&gt; logs from all members covering the time of the reported event?&lt;/p&gt;</comment>
                            <comment id="4153313" author="JIRAUSER1269666" created="Wed, 27 Oct 2021 23:25:03 +0000"  >&lt;p&gt;Hi Dmitry, we only have the mongod log at this moment, since we deleted the whole data directory and sync the db from zero.&lt;/p&gt;</comment>
                            <comment id="4151638" author="dmitry.agranat" created="Wed, 27 Oct 2021 14:51:14 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=zijun.tian%40tusimple.ai&quot; class=&quot;user-hover&quot; rel=&quot;zijun.tian@tusimple.ai&quot;&gt;zijun.tian@tusimple.ai&lt;/a&gt;, in order to understand what has happened during the reported event, we&apos;ll need to review &lt;tt&gt;mongoD&lt;/tt&gt; logs and &lt;tt&gt;diagnostic.data&lt;/tt&gt; from all members of this replica set covering the time of the incident. Please let us know if you still have this data and we&apos;ll provide you with a link to a secure uploader.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="1901817">SERVER-60803</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 27 Oct 2021 14:51:14 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 14 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 14 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>dmitry.agranat@mongodb.com</customfieldvalue>
            <customfieldvalue>zijun.tian@tusimple.ai</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i06bdz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hzpymf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;ol&gt;
	&lt;li&gt;The primary node is down with some data unable to sync to secondary nodes.&lt;/li&gt;
	&lt;li&gt;Some new data writes to the new primary node and sync to the rest of the replica set.&lt;/li&gt;
	&lt;li&gt;Restart the former primary node.&lt;/li&gt;
&lt;/ol&gt;
</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[dmitry.agranat@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i05xjb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>