<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:43:50 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-58184] Checkpoint thread causes assertions when raced with recovering prepared transactions on startup</title>
                <link>https://jira.mongodb.org/browse/SERVER-58184</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;The checkpoint thread reads at the stable timestamp to evaluate the amount of oplog necessary for rollback. If a checkpoint is taken during server startup or after a rollback when we are reconstructing prepared transactions, it may be possible to hit an assertion like this in WiredTiger:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;WT_SESSION.prepare_transaction: __txn_assert_after_reads, 516: prepare timestamp (1623940678, 408) must be greater than the latest active read timestamp (1623940808, 102) : Invalid argument&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;We have only seen this problem reproduce on the code coverage builder, which is extremely slow, and no users have seen this. This has also only been reproduced on 4.4, but it seems like it should affect every version from 4.2 to 5.1.&lt;/p&gt;

&lt;p&gt;A workaround may be to take a global X lock while reconstructing prepared transactions to conflict with the checkpoint thread.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1804612">SERVER-58184</key>
            <summary>Checkpoint thread causes assertions when raced with recovering prepared transactions on startup</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="pavithra.vetriselvan@mongodb.com">Pavithra Vetriselvan</assignee>
                                    <reporter username="louis.williams@mongodb.com">Louis Williams</reporter>
                        <labels>
                    </labels>
                <created>Wed, 30 Jun 2021 18:59:53 +0000</created>
                <updated>Sun, 29 Oct 2023 21:51:27 +0000</updated>
                            <resolved>Thu, 19 Aug 2021 21:30:23 +0000</resolved>
                                    <version>4.4.6</version>
                                    <fixVersion>5.0.3</fixVersion>
                    <fixVersion>4.4.9</fixVersion>
                    <fixVersion>5.1.0-rc0</fixVersion>
                                                        <votes>0</votes>
                                    <watches>10</watches>
                                                                                                                <comments>
                            <comment id="4107731" author="JIRAUSER1259052" created="Wed, 6 Oct 2021 18:34:42 +0000"  >&lt;p&gt;Updating the fixversion since branching activities occurred yesterday. This ticket will be in rc0 when it&#8217;s been triggered. For more active release information, please keep an eye on #server-release. Thank you!&lt;/p&gt;</comment>
                            <comment id="4014082" author="xgen-internal-githook" created="Fri, 20 Aug 2021 18:44:06 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Pavi Vetriselvan&apos;, &apos;email&apos;: &apos;pavithra.vetriselvan@mongodb.com&apos;, &apos;username&apos;: &apos;pvselvan&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-58184&quot; title=&quot;Checkpoint thread causes assertions when raced with recovering prepared transactions on startup&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-58184&quot;&gt;&lt;del&gt;SERVER-58184&lt;/del&gt;&lt;/a&gt; Move getOldestActiveTimestamp after prepare reconstruction&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 841cff317bf34a320f9b8be24cdf27faf4393cbb)&lt;br/&gt;
Branch: v4.4&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/c428d99ef2aa746acd85faee8743c5a67356b714&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/c428d99ef2aa746acd85faee8743c5a67356b714&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4014075" author="xgen-internal-githook" created="Fri, 20 Aug 2021 18:42:21 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Pavi Vetriselvan&apos;, &apos;email&apos;: &apos;pavithra.vetriselvan@mongodb.com&apos;, &apos;username&apos;: &apos;pvselvan&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-58184&quot; title=&quot;Checkpoint thread causes assertions when raced with recovering prepared transactions on startup&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-58184&quot;&gt;&lt;del&gt;SERVER-58184&lt;/del&gt;&lt;/a&gt; Move getOldestActiveTimestamp after prepare reconstruction&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 841cff317bf34a320f9b8be24cdf27faf4393cbb)&lt;br/&gt;
Branch: v5.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/0540ed3053add3e352dc81c6fb7f6dd3a11afc31&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/0540ed3053add3e352dc81c6fb7f6dd3a11afc31&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4010563" author="xgen-internal-githook" created="Thu, 19 Aug 2021 18:07:48 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Pavi Vetriselvan&apos;, &apos;email&apos;: &apos;pavithra.vetriselvan@mongodb.com&apos;, &apos;username&apos;: &apos;pvselvan&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-58184&quot; title=&quot;Checkpoint thread causes assertions when raced with recovering prepared transactions on startup&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-58184&quot;&gt;&lt;del&gt;SERVER-58184&lt;/del&gt;&lt;/a&gt; Move getOldestActiveTimestamp after prepare reconstruction&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/841cff317bf34a320f9b8be24cdf27faf4393cbb&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/841cff317bf34a320f9b8be24cdf27faf4393cbb&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3911048" author="louis.williams" created="Wed, 30 Jun 2021 21:00:44 +0000"  >&lt;p&gt;I just discussed with &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=samy.lanka&quot; class=&quot;user-hover&quot; rel=&quot;samy.lanka&quot;&gt;samy.lanka&lt;/a&gt; and we think that moving the callback registration after reconstruction (the call to &lt;a href=&quot;https://github.com/mongodb/mongo/blob/8659dd3dc2b906d10d9cd5fdb213f7169ec71678/src/mongo/db/mongod_main.cpp#L645&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;replCoord-&amp;gt;startup&lt;/a&gt;) should work. The oplog will not be truncated until that callback is registered. &lt;a href=&quot;https://github.com/mongodb/mongo/blob/11804e1aa48e88ebd65a11eb81f069ebae23ef7b/src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp#L518-L520&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;See here&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="3911013" author="daniel.gottlieb@10gen.com" created="Wed, 30 Jun 2021 20:42:28 +0000"  >&lt;p&gt;Ah thanks for the explanation. My head wasn&apos;t on right, I thought we were preparing things as part of the oplog application step in startup recovery. It makes sense that we&apos;re hitting this because we&apos;re re-preparing arbitrarily old transactions.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;2. Good idea. I had also considered that, but my concern was that maybe some of the steps afterward depend on the &quot;oplog needed for rollback&quot; logic.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;The only bit that depends on that (I hope) is that we don&apos;t truncate any data that&apos;s needed for oplog recovery. We should be &quot;conservative&quot; today and not truncate anything until after that function is registered.&lt;/p&gt;

&lt;p&gt;I&apos;m a little concerned about the MODE_X lock given the FTDC gaps that can create (I&apos;m assuming FTDC is turned on at this point) and there have been a couple complaints lately regarding how little visibility we have into the system at startup and shutdown.&lt;/p&gt;</comment>
                            <comment id="3910975" author="samy.lanka" created="Wed, 30 Jun 2021 20:29:54 +0000"  >&lt;p&gt;My understanding of why we don&apos;t hit this during steady state replication is that we shouldn&apos;t be preparing a transaction at a timestamp before the timestamp that the checkpoint thread reads at (because the stable timestamp wouldn&apos;t have advanced if we had reserved an oplog slot for prepare). We need the out-of-order prepare that happens during reconstructing prepared transactions in recovery. &lt;/p&gt;</comment>
                            <comment id="3910964" author="louis.williams" created="Wed, 30 Jun 2021 20:25:46 +0000"  >&lt;p&gt;1. My guess is that during steady-state we are using a prepare timestamp that doesn&apos;t overlap with the stable timestamp. But I don&apos;t know how that&apos;s guaranteed.&lt;br/&gt;
2. Good idea. I had also considered that, but my concern was that maybe some of the steps afterward depend on the &quot;oplog needed for rollback&quot; logic.&lt;/p&gt;</comment>
                            <comment id="3910946" author="daniel.gottlieb@10gen.com" created="Wed, 30 Jun 2021 20:19:47 +0000"  >&lt;p&gt;Two questions:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Why won&apos;t we hit this during steady state replication?&lt;/li&gt;
	&lt;li&gt;Could we move &lt;a href=&quot;https://github.com/mongodb/mongo/blob/master/src/mongo/db/mongod_main.cpp#L660-L663&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;the registration of the callback&lt;/a&gt; until after recovering prepared transactions?&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="3910849" author="louis.williams" created="Wed, 30 Jun 2021 19:39:34 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=samy.lanka&quot; class=&quot;user-hover&quot; rel=&quot;samy.lanka&quot;&gt;samy.lanka&lt;/a&gt; or &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.gottlieb&quot; class=&quot;user-hover&quot; rel=&quot;daniel.gottlieb&quot;&gt;daniel.gottlieb&lt;/a&gt;, do you have any sense of whether or not my idea is terrible or not? If we consider introducing a different concurrency control mechanism to stop checkpoints while recovering prepared transactions, I think the global X lock looks pretty attractive. Keep in mind we probably need to backport this to 4.2.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="700955">SERVER-39679</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>10.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="21777"><![CDATA[v5.0]]></customfieldvalue>
    <customfieldvalue key="18953"><![CDATA[v4.4]]></customfieldvalue>
    <customfieldvalue key="16775"><![CDATA[v4.2]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 30 Jun 2021 20:19:47 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 18 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 18 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>120.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>daniel.gottlieb@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>louis.williams@mongodb.com</customfieldvalue>
            <customfieldvalue>pavithra.vetriselvan@mongodb.com</customfieldvalue>
            <customfieldvalue>samy.lanka@mongodb.com</customfieldvalue>
            <customfieldvalue>vivian.ge@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hzpuf3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr2zun:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="5037">Repl 2021-07-12</customfieldvalue>
    <customfieldvalue id="5038">Repl 2021-07-26</customfieldvalue>
    <customfieldvalue id="5116">Execution Team 2021-08-09</customfieldvalue>
    <customfieldvalue id="5117">Execution Team 2021-08-23</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hzpgo7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>