<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:12:08 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-46678] Preserve durable history across restarts</title>
                <link>https://jira.mongodb.org/browse/SERVER-46678</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Today, on startup, MongoDB sets the oldest timestamp in such a way as to cause WiredTiger to remove all existing durable history in the history file.  This ticket is to change that behavior such that the history in the history file is preserved on startup.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1240360">SERVER-46678</key>
            <summary>Preserve durable history across restarts</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="daniel.gottlieb@mongodb.com">Daniel Gottlieb</assignee>
                                    <reporter username="milkie@mongodb.com">Eric Milkie</reporter>
                        <labels>
                            <label>PM-234-M3</label>
                            <label>PM-234-T-data-clone</label>
                    </labels>
                <created>Fri, 6 Mar 2020 17:38:51 +0000</created>
                <updated>Sun, 29 Oct 2023 22:11:10 +0000</updated>
                            <resolved>Fri, 8 Jan 2021 20:02:34 +0000</resolved>
                                                    <fixVersion>4.9.0</fixVersion>
                                    <component>Storage</component>
                                        <votes>0</votes>
                                    <watches>11</watches>
                                                                                                                <comments>
                            <comment id="3555609" author="xgen-internal-githook" created="Fri, 8 Jan 2021 19:56:28 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Daniel Gottlieb&apos;, &apos;email&apos;: &apos;daniel.gottlieb@mongodb.com&apos;, &apos;username&apos;: &apos;dgottlieb&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-46678&quot; title=&quot;Preserve durable history across restarts&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-46678&quot;&gt;&lt;del&gt;SERVER-46678&lt;/del&gt;&lt;/a&gt;: Utilize durable history across restarts.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/ca040c1f469aa0ffd68e7a0605c10145e7fb65dc&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/ca040c1f469aa0ffd68e7a0605c10145e7fb65dc&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3486192" author="daniel.gottlieb@10gen.com" created="Wed, 11 Nov 2020 20:17:02 +0000"  >&lt;p&gt;I had a talk with &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=geert.bosch&quot; class=&quot;user-hover&quot; rel=&quot;geert.bosch&quot;&gt;geert.bosch&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=louis.williams&quot; class=&quot;user-hover&quot; rel=&quot;louis.williams&quot;&gt;louis.williams&lt;/a&gt;. We believe that by sacrificing precision of the minimum visible timestamp, we have a relatively low effort low risk way starting up with legal values for the minimum visible timestamp (i.e: no new data needs to be written out, eliminating any upgrade/downgrade work).&lt;/p&gt;

&lt;p&gt;Today on startup, the oldest and stable timestamp are set to the checkpoints recovery timestamp (effectively the stable timestamp as of that checkpoint). Today at startup, &lt;a href=&quot;https://github.com/mongodb/mongo/blob/7728778a2138dddb951ab87e95c4337d8e653a86/src/mongo/db/catalog/catalog_control.cpp#L201-L214&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;none of the collections will have a minVisibleTimestamp&lt;/a&gt; set. The proposed algorithm would be:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;On startup, query the oldest timestamp (which is now populated via &lt;a href=&quot;https://jira.mongodb.org/browse/WT-6331&quot; title=&quot;Set oldest timestamp on startup of WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-6331&quot;&gt;&lt;del&gt;WT-6331&lt;/del&gt;&lt;/a&gt;)&lt;/li&gt;
	&lt;li&gt;Perform a read on the &lt;tt&gt;_mdb_catalog&lt;/tt&gt; with a &lt;tt&gt;read_timestamp&lt;/tt&gt; of the oldest timestamp.&lt;/li&gt;
	&lt;li&gt;Note all collection UUIDs in that read.&lt;/li&gt;
	&lt;li&gt;When constructing Collection objects, we will set the minimum visible timestamp to one of two values:
	&lt;ul&gt;
		&lt;li&gt;For collections who&apos;s UUID was found in a read at the &lt;tt&gt;oldest_timestamp&lt;/tt&gt;, the minimum visible timestamp will remain unset. New readers will be able to do a historical read after a restart and see these collections.&lt;/li&gt;
		&lt;li&gt;For collections who&apos;s UUID was not* found, they&apos;re minimum visible timestamp will be set to the recovery timestamp (the timestamp of the checkpoint the process is starting against).&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
	&lt;li&gt;For indexes, the &lt;tt&gt;_id&lt;/tt&gt; index&apos;s minimum visible timestamp will remain unset for collections available at the &lt;tt&gt;oldest_timestamp&lt;/tt&gt;.&lt;/li&gt;
	&lt;li&gt;All other (secondary) indexes&apos; minimum visible timestamps will be set to the recovery timestamp.&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="3345024" author="milkie" created="Tue, 18 Aug 2020 14:28:00 +0000"  >&lt;p&gt;At the moment, WiredTiger now does preserve history across restarts, but MongoDB sets its oldest timestamp at startup to clear out all preserved history.  In order to not clear out preserved history at startup, MongoDB will need to change how we do minimum visible timestamps for collections, as Max and Dan have mentioned in the above comment.&lt;/p&gt;</comment>
                            <comment id="3344904" author="daniel.gottlieb@10gen.com" created="Tue, 18 Aug 2020 13:42:46 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=max.hirschhorn&quot; class=&quot;user-hover&quot; rel=&quot;max.hirschhorn&quot;&gt;max.hirschhorn&lt;/a&gt; noticed that because the minimum visible timestamp is not preserved across restart, it may not be correct for MDB to simply use the oldest timestamp that WT provides. I think he&apos;s right &amp;#8211; we&apos;ll need catalog versioning (or persistence of the minimum visible timestamp/index build completion times).&lt;/p&gt;</comment>
                            <comment id="3028690" author="vamsi.krishna" created="Tue, 7 Apr 2020 05:24:46 +0000"  >&lt;p&gt;We had discussed this earlier and it is captured in &lt;a href=&quot;https://jira.mongodb.org/browse/WT-5539&quot; title=&quot;Checkpoint doesn&amp;#39;t save the history window that is open during checkpoint&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-5539&quot;&gt;&lt;del&gt;WT-5539&lt;/del&gt;&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/browse/WT-5679&quot; title=&quot;Set the oldest history store checkpoint timestamp as oldest timestamp during restart&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-5679&quot;&gt;&lt;del&gt;WT-5679&lt;/del&gt;&lt;/a&gt;. No work is done, I have scheduled &lt;a href=&quot;https://jira.mongodb.org/browse/WT-5679&quot; title=&quot;Set the oldest history store checkpoint timestamp as oldest timestamp during restart&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-5679&quot;&gt;&lt;del&gt;WT-5679&lt;/del&gt;&lt;/a&gt; in current sprint.&lt;/p&gt;</comment>
                            <comment id="3026870" author="alexander.gorrod" created="Mon, 6 Apr 2020 09:29:46 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=haribabu.kommi&quot; class=&quot;user-hover&quot; rel=&quot;haribabu.kommi&quot;&gt;haribabu.kommi&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=vamsi.krishna&quot; class=&quot;user-hover&quot; rel=&quot;vamsi.krishna&quot;&gt;vamsi.krishna&lt;/a&gt; we talked about introducing a mechanism to facilitate this, but I don&apos;t remember how far we got. I &lt;b&gt;think&lt;/b&gt; we were going to remember the oldest timestamp serviced by each checkpoint, and find the oldest global checkpoint. Did we do that work? If so is there a simple way we could expose the oldest available timestamp for reads after a restart?&lt;/p&gt;</comment>
                            <comment id="3025107" author="daniel.gottlieb@10gen.com" created="Fri, 3 Apr 2020 14:53:40 +0000"  >&lt;p&gt;The last time I looked into this, WT does not track what a legal &lt;tt&gt;oldest_timestamp&lt;/tt&gt; is across restarts. As in, a WT program can set oldest + stable timestamp to 100, restart, set the oldest timestamp to 50 and perform a read at time 50 and possibly be returned wrong data instead of an error. In the absence of the application writing down the oldest timestamps it has informed WT of, the application must reset the &lt;tt&gt;oldest_timestamp&lt;/tt&gt; to the restarted data&apos;s &lt;tt&gt;recovery timestamp&lt;/tt&gt; (stable timestamp at shutdown).&lt;/p&gt;

&lt;p&gt;A MongoDB-only change to preserve history across restarts would probably be of the form:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Each time the oldest timestamp is about to be updated:
	&lt;ul&gt;
		&lt;li&gt;Write the new value to disk&lt;/li&gt;
		&lt;li&gt;Inform WT of the new value&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;On restart, the value read from disk is guaranteed to be a valid &lt;tt&gt;oldest_timestamp&lt;/tt&gt;. The corollary solution in WT would be:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Each time WT is about to vacuum some data out of the history store (up through potentially time &lt;tt&gt;T&lt;/tt&gt;, the oldest timestamp?):
	&lt;ul&gt;
		&lt;li&gt;Write &lt;tt&gt;T&lt;/tt&gt; to disk&lt;/li&gt;
		&lt;li&gt;Start the vacuuming process&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Today, MongoDB updates the oldest timestamp very frequently. I expect MongoDB updates the oldest timestamp much more frequently than WT vacuums history. I suspect the proposed MongoDB algorithm would cause problems due to excessive (albeit small) writes to disk. Alternatively, MongoDB could slow down how often it informs WT of a new oldest timestamp (reduces the writes MongoDB makes, but limits WTs ability to batch/optimize its vacuuming process).&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=alexander.gorrod&quot; class=&quot;user-hover&quot; rel=&quot;alexander.gorrod&quot;&gt;alexander.gorrod&lt;/a&gt; is there an existing WT ticket aimed at protecting users against setting the oldest timestamp across restarts to an illegal value?&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="1366267">WT-6331</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is depended on by">
                                        <issuelink>
            <issuekey id="1577151">SERVER-53516</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 3 Apr 2020 14:53:40 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 4 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/WT-6331'>WT-6331</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-234</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 4 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>alexander.gorrod@mongodb.com</customfieldvalue>
            <customfieldvalue>daniel.gottlieb@mongodb.com</customfieldvalue>
            <customfieldvalue>milkie@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>vamsi.krishna@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hx2vt3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hxznyn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3633">Execution Team 2020-04-06</customfieldvalue>
    <customfieldvalue id="4214">Execution Team 2020-10-05</customfieldvalue>
    <customfieldvalue id="4215">Execution Team 2020-10-19</customfieldvalue>
    <customfieldvalue id="4259">Sharding 2020-12-28</customfieldvalue>
    <customfieldvalue id="4296">Execution Team 2020-11-02</customfieldvalue>
    <customfieldvalue id="4297">Execution Team 2020-11-16</customfieldvalue>
    <customfieldvalue id="4482">Sharding 2021-01-11</customfieldvalue>
    <customfieldvalue id="4483">Sharding 2021-01-25</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hx2i2f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>