<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:37:24 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-55766] Introduce an optimized &quot;for restore&quot; startup replication recovery mechanism</title>
                <link>https://jira.mongodb.org/browse/SERVER-55766</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;After a restore users generally don&apos;t need to be able to roll back or do PIT reads earlier than the top of the oplog.&lt;/p&gt;

&lt;p&gt;Replication recovery can also be very long after a restore, and the stable/oldest timestamp cannot advance during replication recovery. This isn&apos;t great even with durable history, but can lead to very poor performance in 4.2 before durable history. &lt;/p&gt;

&lt;p&gt;We should provide a startup parameter, that when configured, applies oplog entries either:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;without timestamps to create no history, or&lt;/li&gt;
	&lt;li&gt;with timestamps, but advancing the stable/oldest timestamp between batches&lt;br/&gt;
so that the storage engine can evict history. &lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;We may have to set the initial data timestamp at the end of recovery to prevent rollbacks or reads before the timestamp at the end of recovery. We also need to consider what happens when the nodes crashes halfway through recovery, and make sure it doesn&apos;t corrupt data in that case.&lt;/p&gt;

&lt;p&gt;This should only be supported and used in Atlas.&lt;/p&gt;

&lt;p&gt;Note that if a rollback were necessary to a point before the end of the recovery, the rollback would fail unrecoverably. If the restore was used to seed a new replica set, it is not expected that a node in that set would roll back to a point before the last seeded oplog entry.&lt;/p&gt;

&lt;p&gt;Credit to &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=lingzhi.deng&quot; class=&quot;user-hover&quot; rel=&quot;lingzhi.deng&quot;&gt;lingzhi.deng&lt;/a&gt; for this idea.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1669922">SERVER-55766</key>
            <summary>Introduce an optimized &quot;for restore&quot; startup replication recovery mechanism</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="matthew.russotto@mongodb.com">Matthew Russotto</assignee>
                                    <reporter username="judah.schvimer@mongodb.com">Judah Schvimer</reporter>
                        <labels>
                    </labels>
                <created>Fri, 2 Apr 2021 19:42:51 +0000</created>
                <updated>Sun, 29 Oct 2023 21:55:23 +0000</updated>
                            <resolved>Mon, 3 May 2021 15:34:37 +0000</resolved>
                                                    <fixVersion>4.2.15</fixVersion>
                    <fixVersion>4.4.7</fixVersion>
                    <fixVersion>5.0.0-rc0</fixVersion>
                                    <component>Replication</component>
                                        <votes>1</votes>
                                    <watches>21</watches>
                                                                                                                <comments>
                            <comment id="3769268" author="xgen-internal-githook" created="Wed, 12 May 2021 15:12:28 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Matthew Russotto&apos;, &apos;email&apos;: &apos;matthew.russotto@mongodb.com&apos;, &apos;username&apos;: &apos;mtrussotto&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-55766&quot; title=&quot;Introduce an optimized &amp;quot;for restore&amp;quot; startup replication recovery mechanism&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-55766&quot;&gt;&lt;del&gt;SERVER-55766&lt;/del&gt;&lt;/a&gt; Introduce an optimized &quot;for restore&quot; startup replication recovery mechanism&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 40b7635321caadb7219f7d990a049a93d9776490)&lt;br/&gt;
Branch: v4.2&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/2a99e03b813f33342ffe83ccc5df9b8d2c33bf08&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/2a99e03b813f33342ffe83ccc5df9b8d2c33bf08&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3766438" author="xgen-internal-githook" created="Tue, 11 May 2021 14:22:17 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Matthew Russotto&apos;, &apos;email&apos;: &apos;matthew.russotto@mongodb.com&apos;, &apos;username&apos;: &apos;mtrussotto&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-55766&quot; title=&quot;Introduce an optimized &amp;quot;for restore&amp;quot; startup replication recovery mechanism&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-55766&quot;&gt;&lt;del&gt;SERVER-55766&lt;/del&gt;&lt;/a&gt; Introduce an optimized &quot;for restore&quot; startup replication recovery mechanism&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 8a7e9a21fd0e10ddc1b41345e5bea1a82141061b)&lt;br/&gt;
Branch: v4.4&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/40b7635321caadb7219f7d990a049a93d9776490&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/40b7635321caadb7219f7d990a049a93d9776490&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3747691" author="xgen-internal-githook" created="Thu, 29 Apr 2021 21:07:15 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Matthew Russotto&apos;, &apos;email&apos;: &apos;matthew.russotto@mongodb.com&apos;, &apos;username&apos;: &apos;mtrussotto&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-55766&quot; title=&quot;Introduce an optimized &amp;quot;for restore&amp;quot; startup replication recovery mechanism&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-55766&quot;&gt;&lt;del&gt;SERVER-55766&lt;/del&gt;&lt;/a&gt; Introduce an optimized &quot;for restore&quot; startup replication recovery mechanism&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/8a7e9a21fd0e10ddc1b41345e5bea1a82141061b&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/8a7e9a21fd0e10ddc1b41345e5bea1a82141061b&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3702752" author="judah.schvimer" created="Tue, 6 Apr 2021 13:10:04 +0000"  >&lt;p&gt;If recovering safely after a crash mid-recovery in &quot;for restore&quot;-mode were difficult, we could maybe require that the entire restore process must succeed in one go, possibly by setting the initial sync flag at the beginning of recovery.&lt;/p&gt;

&lt;p&gt;I suspect we don&apos;t actually want to advance the stable timestamp during recovery, since that would be a &quot;lie&quot; about the majority commit point. After transitioning out of recovery, correcting this lie would be difficult. Rather, we would want to set the &lt;tt&gt;InitialDataTimestamp&lt;/tt&gt; to &lt;tt&gt;Timestamp::kAllowUnstableCheckpointsSentinel&lt;/tt&gt;. This would ensure we take unstable checkpoints periodically during recovery that allow the storage engine to evict cache and store less history. We have code for recovering from an unstable checkpoint, which I think is (only?) used when a node has the &lt;tt&gt;takeUnstableCheckpointOnShutdown&lt;/tt&gt; flag set.&lt;/p&gt;</comment>
                            <comment id="3699326" author="milkie" created="Fri, 2 Apr 2021 20:04:58 +0000"  >&lt;p&gt;I&apos;m not sure option 1 will be very effective, as the untimestamped writes will still create history, it&apos;s just complicated how they work and there might still be bugs with how they are handled for durable history.  I think we should pursue option 2 first when we implement this.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10520">
                    <name>Problem/Incident</name>
                                            <outwardlinks description="causes">
                                        <issuelink>
            <issuekey id="2459807">SERVER-81878</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2460039">SERVER-81879</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="1657845">SERVER-55483</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="18953"><![CDATA[v4.4]]></customfieldvalue>
    <customfieldvalue key="16775"><![CDATA[v4.2]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[5002K00000tV4bkQAC]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 2 Apr 2021 20:04:58 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 39 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_17052" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Downstream Changes Summary</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Introduces new setParameter &amp;#39;startupRecoveryForRestore&amp;#39; which should be used along with &amp;#39;recoverFromOplogAsStandalone&amp;#39; and &amp;#39;takeUnstableCheckpointAtShutdown&amp;#39; when doing point-in-time restores.</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16942"><![CDATA[Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 39 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>milkie@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>judah.schvimer@mongodb.com</customfieldvalue>
            <customfieldvalue>matthew.russotto@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hz31x3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hyo5if:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="4858">Repl 2021-05-03</customfieldvalue>
    <customfieldvalue id="4859">Repl 2021-05-17</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_17051" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Teams Impacted</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16943"><![CDATA[Cloud]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hz2o67:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>