<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:40:55 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-35774] Initial Sync not replicating old oplog entries may have a stale node give up resync and permanently stay in RECOVERING state</title>
                <link>https://jira.mongodb.org/browse/SERVER-35774</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;The following is the shortest schedule that can manifest the bug&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&amp;#8212;&lt;/p&gt;

&lt;p&gt;Environment Setup:&lt;/p&gt;

&lt;p&gt;Environment is where I have 2 TB hard disk drives and ran all server nodes on the same host. This seems cause slow startup time on some nodes. (Creating preallocated &apos;local&apos; files seems to take long time &#8211; few minutes). OS is Ubuntu 16.04. I used MongoDB version 3.0.0. &#160;Also, I configured to turn off the chaining.&lt;/p&gt;

&lt;p&gt;Initially, start three nodes. Say node 1,2 and 3. Create 3 documents initialized as {&quot;_id&quot; : &quot;key0&quot;, &quot;value&quot; : &quot;0&quot;},&#160; {&quot;_id&quot; : &quot;key1&quot;, &quot;value&quot; : &quot;1&quot;} , {&quot;_id&quot; : &quot;key2&quot;,&#160; &#160;&quot;value&quot; : &quot;2&quot; }, respectively.&lt;/p&gt;

&lt;p&gt;&amp;#8212;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&amp;#8212;&lt;/p&gt;

&lt;p&gt;Then, do following steps:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Assuming node 0 is the primary, crash node 1 and 2. Then, invoke multiple updateOne to repeatedly update the document with &quot;_id&quot; : &quot;key0&quot; to incrementally set &quot;value&quot; : &quot;1000.1&quot;, &quot;value&quot; : &quot;1000.2&quot; ... &quot;value&quot; : &quot;1000.9&quot;. Crash node 0.&lt;/li&gt;
	&lt;li&gt;Restart node 0 and 1. Wait for their optime get converged.&lt;/li&gt;
	&lt;li&gt;Crash node 1. Then, invoke multiple updateOne to repeatedly update the document with &quot;_id&quot; : &quot;key1&quot; to incrementally set &quot;value&quot; : &quot;1001.1&quot;, &quot;value&quot; : &quot;1001.2&quot; ... &quot;value&quot; : &quot;1001.9&quot;. Crash node 0.&lt;/li&gt;
	&lt;li&gt;Restart node 0 and 2. Wait for their optime get converged.&lt;/li&gt;
	&lt;li&gt;Crash node 2. Then, invoke multiple updateOne to repeatedly update the document with &quot;_id&quot; : &quot;key2&quot; to incrementally set &quot;value&quot; : &quot;1002.1&quot;, &quot;value&quot; : &quot;1002.2&quot; ... &quot;value&quot; : &quot;1002.9&quot;. Crash node 0.&lt;/li&gt;
	&lt;li&gt;Restart node 1 and 2. Wait for their optime get converged.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;&amp;#8212;&lt;/p&gt;

&lt;p&gt;At the step 2, node 1 bumps up its optime &amp;#8211; say 10. At the step 4, node 2 bumps up its optime --say 20. Note that both node 1 and node 2 did resync with node 0 via initial sync because oplog was empty. Oplog is empty on both node 1 and 2 because replication has never made from node 0 due to slow startup time of node 1 and 2 during the initialization phase. ( By inspecting log messages, figured that they ran very slowly and never get a chance to replicate even the initial create commands.)&lt;/p&gt;

&lt;p&gt;Based on my source code reading, the function _initialSync() in rs_initialsync.cpp just clones the data and only add the oplog entry for the last operation for the corresponding collection. Hence, after the step 2, node 1 only has the oplog entry for the update at optime 10. After the step 4, node 2 only has the oplog entry for the update at optime 20. They do not have older oplog entries.&lt;/p&gt;

&lt;p&gt;As a result, at the step 6, the node 2 has the oldest oplog entry at optime 20. Yet, the node 1 has the last optime at 10. Thus, the node 1 refuses to resync with the node 2 because there are no older oplog entries &amp;#8211; entries at optime 11 to 19. &#160;This is no problem if the oplog on node 1 was empty because it will fall back to initial sync. However, since the node 1 has the some oplog entry, it does not fall back to initial sync.&lt;/p&gt;

&lt;p&gt;Therefore, the node 1 permanently stays in RECOVERING state. This will block convergence and clients cannot read data from the node 1 indefinitely. It requires manual effort to explicitly use other utilities to trigger initial sync. Doing an initial sync is expensive, and probably that is why MongoDB is designed to be hung instead of automatically triggering the initial sync in such a case we discussed above.&lt;/p&gt;

&lt;p&gt;Yet, to reduce the chance of falling into the situation where we need to go through the expensive initial sync, I think we should replicate some number of older oplog entries even during the initial sync.&lt;/p&gt;</description>
                <environment></environment>
        <key id="563772">SERVER-35774</key>
            <summary>Initial Sync not replicating old oplog entries may have a stale node give up resync and permanently stay in RECOVERING state</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="nick.brewer">Nick Brewer</assignee>
                                    <reporter username="beomheyn.kim@gmail.com">anaud anaud</reporter>
                        <labels>
                    </labels>
                <created>Mon, 25 Jun 2018 15:10:38 +0000</created>
                <updated>Thu, 6 Apr 2023 16:15:03 +0000</updated>
                            <resolved>Mon, 25 Jun 2018 17:59:49 +0000</resolved>
                                    <version>3.0.0</version>
                                                    <component>Internal Code</component>
                                        <votes>0</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="1930846" author="nick.brewer" created="Mon, 25 Jun 2018 17:59:34 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kaelus&quot; class=&quot;user-hover&quot; rel=&quot;kaelus&quot;&gt;kaelus&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;MongoDB 3.0 is no longer supported per our &lt;a href=&quot;https://www.mongodb.com/support-policy&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;support policy&lt;/a&gt;.&#160;I&apos;d suggest upgrading to a supported version.&#160;&lt;/p&gt;

&lt;p&gt;Regards,&#160;&lt;/p&gt;

&lt;p&gt;Nick&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 25 Jun 2018 16:16:50 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 33 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 33 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>beomheyn.kim@gmail.com</customfieldvalue>
            <customfieldvalue>nick.brewer</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hu1cxr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hts5pj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hu0z73:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>