<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:20:47 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-49781] Setting lastApplied before startup recovery has finished causes race with reconstructing prepared transactions</title>
                <link>https://jira.mongodb.org/browse/SERVER-49781</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;When preparing a transaction, WT requires that the prepare timestamp is greater than the latest active read timestamp. To allow for this, we often expect readers to do untimestamped reads during startup to avoid conflicting with prepared transactions (which are only reconstructed at the end of recovery). One way we achieved this in the past was by only setting lastApplied at the end of recovery, so that anything that reads with a kLastApplied read source does an untimestamped read instead.&lt;/p&gt;

&lt;p&gt;However, we now &lt;a href=&quot;https://github.com/mongodb/mongo/blob/92bab3bd50160122b7b5ecaa4126c43fc6bab6d6/src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp#L717&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;set lastApplied on the snapshot manager&lt;/a&gt; before finishing recovery (and specifically before reconstructing prepared transactions). We do this to prevent a race with kNoOverlap that could cause a reader to see writes that shouldn&apos;t be visible. Ultimately, this means that operations reading with a kLastApplied read source during startup can race with reconstructing prepared transactions.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1416196">SERVER-49781</key>
            <summary>Setting lastApplied before startup recovery has finished causes race with reconstructing prepared transactions</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="louis.williams@mongodb.com">Louis Williams</assignee>
                                    <reporter username="samy.lanka@mongodb.com">Samyukta Lanka</reporter>
                        <labels>
                    </labels>
                <created>Tue, 21 Jul 2020 20:27:57 +0000</created>
                <updated>Thu, 24 Jun 2021 00:49:00 +0000</updated>
                            <resolved>Tue, 15 Sep 2020 17:18:33 +0000</resolved>
                                                    <fixVersion>4.4.2</fixVersion>
                                                        <votes>0</votes>
                                    <watches>9</watches>
                                                                                                                <comments>
                            <comment id="3387916" author="louis.williams" created="Fri, 11 Sep 2020 18:22:55 +0000"  >&lt;p&gt;This should be alleviated by &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-48452&quot; title=&quot;Internal readers should default to reading without a timestamp&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-48452&quot;&gt;&lt;del&gt;SERVER-48452&lt;/del&gt;&lt;/a&gt;, but I&apos;m going to keep this ticket open to track my progress on the 4.4 backport.&lt;/p&gt;</comment>
                            <comment id="3299252" author="daniel.gottlieb@10gen.com" created="Thu, 23 Jul 2020 17:14:46 +0000"  >&lt;p&gt;User requests can run during the reconstructing prepared transactions phase of startup/rollback recovery?&lt;/p&gt;</comment>
                            <comment id="3299241" author="louis.williams" created="Thu, 23 Jul 2020 17:08:04 +0000"  >&lt;p&gt;It seems like any user request that reads at kLastApplied could also encounter this problem. The free monitoring service just happened to hit this in our tests. Is that right?&lt;/p&gt;

&lt;p&gt;Would it make sense to take both approaches? That is:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Reject kLastApplied and kNoOverlap reads when neither lastApplied nor all_durable is available&lt;/li&gt;
	&lt;li&gt;Default internal readers to use kNoTimestamp instead of kLastApplied.&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="3297319" author="daniel.gottlieb@10gen.com" created="Wed, 22 Jul 2020 18:25:49 +0000"  >&lt;p&gt;My apologies, I&apos;m familiar with the invariant. I was trying to get more information on what the reader was, because it was omitted I had just assumed the reader was related to reconstructing the prepared updates. &lt;/p&gt;

&lt;p&gt;To me, it&apos;s a bug when any internal reader can have its read source changed due to being in an unset state. My grepping of the &lt;tt&gt;free_mon&lt;/tt&gt; directory doesn&apos;t show any setting of an explicit read source. Can you confirm that the recovery unit on the problematic reader was &lt;a href=&quot;https://github.com/mongodb/mongo/blob/db78b73d0b1cda09831163e9283e7ff5adf90fa7/src/mongo/db/db_raii.cpp#L139&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;simply unset&lt;/a&gt; (which would &lt;a href=&quot;https://github.com/mongodb/mongo/blob/db78b73d0b1cda09831163e9283e7ff5adf90fa7/src/mongo/db/storage/snapshot_helper.cpp#L107-L117&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;opt into last applied behavior&lt;/a&gt; if a &lt;a href=&quot;https://github.com/mongodb/mongo/blob/db78b73d0b1cda09831163e9283e7ff5adf90fa7/src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.cpp#L583-L584&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;timestamp exists&lt;/a&gt;).&lt;/p&gt;</comment>
                            <comment id="3297211" author="samy.lanka" created="Wed, 22 Jul 2020 18:07:31 +0000"  >&lt;blockquote&gt;&lt;p&gt;Can you confirm that the read source for reconstructing prepared transactions is explicitly using kLastApplied&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Sorry for any confusion, the issue isn&apos;t that the read source for reconstructing prepared transactions is using kLastApplied, but that other readers are able to read with kLastApplied even though reconstructing prepared transactions hasn&apos;t happened yet.&lt;/p&gt;

&lt;p&gt;Let&apos;s say that the last entry in the oplog is the prepare entry. Any reader (such as the Free Monitoring Processor) using kLastApplied will be reading at the prepareTimestamp. But since we haven&apos;t reconstructed the transaction yet, when we go to prepare it in WT, we&apos;ll see that there is an active reader at the same timestamp that we want to prepare at, which violates our contract for the prepareTimestamp.&lt;/p&gt;</comment>
                            <comment id="3296750" author="daniel.gottlieb@10gen.com" created="Wed, 22 Jul 2020 15:32:26 +0000"  >&lt;p&gt;&lt;b&gt;Edit&lt;/b&gt;&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Ultimately, this means that operations reading with a kLastApplied read source during startup can race with reconstructing prepared transactions.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;I misread this the first time. Can you confirm that the read source for reconstructing prepared transactions is explicitly using &lt;tt&gt;kLastApplied&lt;/tt&gt;, as opposed to just accidentally picking up the last applied timestamp because the readsource is simply unset? If it is explicitly being set, is there a reason &lt;tt&gt;kLastApplied&lt;/tt&gt; was chosen instead of &lt;tt&gt;kNoTimestamp&lt;/tt&gt;?&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Original comment&lt;/b&gt;&lt;br/&gt;
Setting last applied at startup was also one of the (multiple necessary) changes that resulted in&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-48934&quot; title=&quot;Investigate replica set data mismatch from Jepsen testing&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-48934&quot;&gt;&lt;del&gt;SERVER-48934&lt;/del&gt;&lt;/a&gt;. I&apos;m curious if we should audit other startup&#160;readers to see what is now reading at a last applied value instead of the previous behavior that would end up reading without a timestamp. I believe we&apos;d be able to investigate that programmatically. &lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-48452&quot; title=&quot;Internal readers should default to reading without a timestamp&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-48452&quot;&gt;&lt;del&gt;SERVER-48452&lt;/del&gt;&lt;/a&gt; (internal readers should default to using kNoTimestamp) sounds like it would also avoid this?&lt;/p&gt;</comment>
                            <comment id="3296733" author="milkie" created="Wed, 22 Jul 2020 15:29:27 +0000"  >&lt;p&gt;If there is nothing in the oplog at the end of startup recovery, I believe that corresponds to a system where timestamped writes are not possible (e.g. a replica set node prior to initiate command).  Therefore, we don&apos;t need to do anything for lastApplied in this case, although it wouldn&apos;t hurt to set it to something like initialDataTimestamp.&lt;/p&gt;</comment>
                            <comment id="3295463" author="samy.lanka" created="Tue, 21 Jul 2020 20:38:23 +0000"  >&lt;p&gt;One possible solution that &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=louis.williams&quot; class=&quot;user-hover&quot; rel=&quot;louis.williams&quot;&gt;louis.williams&lt;/a&gt; came up with was to make kNoOverlap reads return SnapshotUnavailable until lastApplied is set. However, we cannot guarantee that lastApplied is set &lt;a href=&quot;https://github.com/mongodb/mongo/blob/0cb70e9577c46257798d0385b15ec6bff8dbd28d/src/mongo/db/repl/replication_coordinator_impl.cpp#L682-L687&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;if there is nothing in the oplog at the end of startup recovery&lt;/a&gt;. Perhaps we could set lastApplied to the initialDataTimestamp in this case?&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="1364002">SERVER-48452</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="1454381">WT-6626</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="994549">SERVER-44529</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1253817">SERVER-46721</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="20506"><![CDATA[v4.7]]></customfieldvalue>
    <customfieldvalue key="18953"><![CDATA[v4.4]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 22 Jul 2020 15:29:27 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 21 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>m.maher@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 21 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>64.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>daniel.gottlieb@mongodb.com</customfieldvalue>
            <customfieldvalue>milkie@mongodb.com</customfieldvalue>
            <customfieldvalue>louis.williams@mongodb.com</customfieldvalue>
            <customfieldvalue>samy.lanka@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxvysv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hw7sjr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="4127">Execution Team 2020-09-07</customfieldvalue>
    <customfieldvalue id="4213">Execution Team 2020-09-21</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxvl27:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>