<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:53:02 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-39762] Fix fastcount after rollback recovery of prepared transactions. </title>
                <link>https://jira.mongodb.org/browse/SERVER-39762</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;When we recover a prepared transaction, we re-apply the oplog entry and replay the operations in the transaction to get back to the prepare state. &lt;a href=&quot;https://github.com/mongodb/mongo/blob/4a0d769e44e6a022e7c129c1beca6a7c86cdd818/src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp#L1707-L1709&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;This&lt;/a&gt; line prevents us from adding a&#160;NumRecordsChange on the new recovery unit.&lt;/p&gt;

&lt;p&gt;During rollback, say we commit a prepared transaction on the rollback node. This entry will get rolled back, the transaction will get invalidated (which releases our txnResources, including the recovery unit), and we will replay oplog entries from the stable timestamp. This involves putting the corresponding prepared transaction back into prepare.&lt;/p&gt;

&lt;p&gt;Since the new recovery unit does not register the operations of the prepared transaction, when we decide to either commit/abort it, there is nothing to add or subtract to numRecords.&lt;/p&gt;

&lt;p&gt;Previously, we have accounted for counts during rollback in&#160;_findRecordStoreCounts, where we calculate the diffs for each UUID. For prepared transactions, however, we are not sure if we will eventually commit or eventually abort the transaction. So, it might not make sense to preemptively change the counts.&lt;/p&gt;

&lt;p&gt;Ideally, we would figure out a way for the new recovery unit to make an exception for prepared transactions that were recovered during rollback and make sure the changes get recorded when reapplying operations.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</description>
                <environment></environment>
        <key id="702920">SERVER-39762</key>
            <summary>Fix fastcount after rollback recovery of prepared transactions. </summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="louis.williams@mongodb.com">Louis Williams</assignee>
                                    <reporter username="pavithra.vetriselvan@mongodb.com">Pavithra Vetriselvan</reporter>
                        <labels>
                            <label>prepare_durability</label>
                            <label>txn_storage</label>
                    </labels>
                <created>Fri, 22 Feb 2019 16:54:35 +0000</created>
                <updated>Sun, 29 Oct 2023 22:23:41 +0000</updated>
                            <resolved>Wed, 27 Mar 2019 19:40:28 +0000</resolved>
                                                    <fixVersion>4.1.10</fixVersion>
                                    <component>Replication</component>
                    <component>Storage</component>
                                        <votes>0</votes>
                                    <watches>5</watches>
                                                                                                                <comments>
                            <comment id="2193655" author="xgen-internal-githook" created="Wed, 27 Mar 2019 19:39:58 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Louis Williams&apos;, &apos;username&apos;: &apos;louiswilliams&apos;, &apos;email&apos;: &apos;louis.williams@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-39762&quot; title=&quot;Fix fastcount after rollback recovery of prepared transactions. &quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-39762&quot;&gt;&lt;del&gt;SERVER-39762&lt;/del&gt;&lt;/a&gt; Fix fastcount after rollback recovery of prepared transactions&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/884f388d3eca3bd5e9fe32f46b3007ccdd89af17&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/884f388d3eca3bd5e9fe32f46b3007ccdd89af17&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2188161" author="max.hirschhorn@10gen.com" created="Thu, 21 Mar 2019 23:02:53 +0000"  >&lt;p&gt;Thanks for the confirmation both of you. I put together&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;212027_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/212027/212027_txn-rollback.png&quot; title=&quot;txn-rollback.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;212027&quot; file-preview-title=&quot;txn-rollback.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/212027/_thumb_212027.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;in order to keep the valid transitions straight in my mind.&lt;/p&gt;</comment>
                            <comment id="2187211" author="louis.williams" created="Thu, 21 Mar 2019 13:57:25 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=max.hirschhorn&quot; class=&quot;user-hover&quot; rel=&quot;max.hirschhorn&quot;&gt;max.hirschhorn&lt;/a&gt; Yes, that is correct. If a node rolls back a &quot;commit&quot; or &quot;abort&quot; back into a prepared state, the sharded transaction protocol guarantees the same command will be sent again; it will not change its mind. If that were not the case, we would need to do some amount of work to make the fastcount logic work differently for rollback.&lt;/p&gt;</comment>
                            <comment id="2187201" author="judah.schvimer" created="Thu, 21 Mar 2019 13:49:39 +0000"  >&lt;blockquote&gt;
&lt;p&gt;Where does the guarantee that a different outcome won&apos;t ever occur on the two branches of history come from? Is it because of how prepared transactions are used by the distributed commit protocol where the coordinator must not change its mind as whether to commit or abort a transaction once it is prepared on all of the shards?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Exactly.&lt;/p&gt;</comment>
                            <comment id="2187062" author="max.hirschhorn@10gen.com" created="Thu, 21 Mar 2019 05:12:24 +0000"  >&lt;blockquote&gt;
&lt;p&gt;&lt;b&gt;rollback &quot;abort&quot; into &quot;prepare&quot;&lt;/b&gt; (guaranteed to eventually abort)&lt;br/&gt;
&lt;b&gt;rollback &quot;commit&quot; into &quot;prepare&quot;&lt;/b&gt; (guaranteed to eventually commit)&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;Where does the guarantee that a different outcome won&apos;t ever occur on the two branches of history come from? Is it because of how prepared transactions are used by the distributed commit protocol where the coordinator must not change its mind as whether to commit or abort a transaction once it is prepared on all of the shards?&lt;/p&gt;

&lt;p&gt;Modeling these semantics is important to me for implementing prepared transaction support in the rollback fuzzer.&lt;/p&gt;</comment>
                            <comment id="2186633" author="louis.williams" created="Wed, 20 Mar 2019 20:34:57 +0000"  >&lt;p&gt;After some discussion, it seems there has been agreement about the desired behavior of fastcount&#160;during rollback. The desire is to do no additional work, only confirm we have testing to cover all cases.&lt;/p&gt;

&lt;p&gt;Keep in mind that we &lt;a href=&quot;https://github.com/mongodb/mongo/blob/98e970f09d154da6190a0808cd846e566438a349/src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp#L1656-L1675&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;update the fastcount&#160;in memory, then rollback if the storage transaction aborts.&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;There are 5 cases to consider:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;&lt;b&gt;rollback &quot;abort&quot; into &quot;prepare&quot;&lt;/b&gt; (guaranteed to eventually abort)
	&lt;ul&gt;
		&lt;li&gt;When a transaction is aborted, the fast count is also rolled back. When &lt;a href=&quot;https://github.com/mongodb/mongo/blob/98e970f09d154da6190a0808cd846e566438a349/src/mongo/db/repl/replication_recovery.cpp#L274&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;the prepared transaction is &quot;reconstructed&quot;&lt;/a&gt; at the end of replication recovery by applying the operations and creating a new prepared storage transaction, the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/4a0d769e44e6a022e7c129c1beca6a7c86cdd818/src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp#L1707-L1709&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;fastcount is not&#160;adjusted&lt;/a&gt;. When the transaction is aborted again, there is nothing to roll back. This leaves the fastcount in the correct state.&lt;/li&gt;
		&lt;li&gt;I have confirmed locally this is true, but I will use this ticket to add assertions.&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
	&lt;li&gt;&lt;b&gt;rollback &quot;abort&quot; and &quot;prepare&quot;&lt;/b&gt; (guaranteed to never commit, but could prepare again)
	&lt;ul&gt;
		&lt;li&gt;When a transaction is aborted, the fast count is also rolled back. Rollback of the &quot;prepare: true&quot; applyOps oplog entry &lt;a href=&quot;https://github.com/mongodb/mongo/blob/4a0d769e44e6a022e7c129c1beca6a7c86cdd818/src/mongo/db/repl/rollback_impl.cpp#L641-L645&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;calculates the fastcount adjustment&lt;/a&gt; to make once recovery is complete, and then also subtracts that from the collection count.&lt;/li&gt;
		&lt;li&gt;&lt;em&gt;I don&apos;t believe this is correct&lt;/em&gt;, so I will use this ticket to investigate and fix this. We may be able to just ignore &quot;prepare: true&quot; oplog entries for consideration of size adjustment.&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
	&lt;li&gt;&lt;b&gt;rollback &quot;commit&quot;&#160;into&#160;&quot;prepare&quot;&lt;/b&gt; (guaranteed to eventually commit)
	&lt;ul&gt;
		&lt;li&gt;When a transaction is committed, the fast count is unmodified. When the prepared transaction is reconstructed, like the first case, there are no fastcount adjustments made, and when the transaction eventually commits, the counts also remain unmodified.&lt;/li&gt;
		&lt;li&gt;I have confirmed locally this is true, but I will use this ticket to add assertions.&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
	&lt;li&gt;&lt;b&gt;rollback &quot;prepare&quot;&lt;/b&gt; (could eventually commit or abort)
	&lt;ul&gt;
		&lt;li&gt;During rollback, prepared transactions are aborted, which rolls back the fastcount. If the same &quot;prepare&quot; operation comes in at a later point, it will just re-apply the same operations as before.&lt;/li&gt;
		&lt;li&gt;See&#160;&lt;b&gt;rollback &quot;abort&quot; and &quot;prepare&quot;,&lt;/b&gt; but I believe this is affected by the same double-counting bug. The abort before rollback resets the fastcount, and then the rollback of the &quot;prepare&quot; entry, subtracts the same counts again.&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
	&lt;li&gt;&lt;b&gt;rollback &quot;commit&quot;&#160;and&#160;&quot;prepare&quot;&lt;/b&gt;
	&lt;ul&gt;
		&lt;li&gt;This is impossible because &quot;prepare&quot; must be majority committed before sending &quot;commit&quot;&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2184451" author="louis.williams" created="Mon, 18 Mar 2019 22:25:08 +0000"  >&lt;p&gt;I don&apos;t think this is going to be as simple as just recording op counts while reconstructing prepare oplog entries. I think we&apos;ll need first locate the prepare oplog entries, calculate the operation counts per namespace, and then subtract those from each collection&apos;s in-memory count.&#160;&lt;/p&gt;</comment>
                            <comment id="2161447" author="judah.schvimer" created="Mon, 25 Feb 2019 14:50:12 +0000"  >&lt;p&gt;Due to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-35483&quot; title=&quot;rollback makes config.transactions fastcount inaccurate&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-35483&quot;&gt;&lt;del&gt;SERVER-35483&lt;/del&gt;&lt;/a&gt;, this won&apos;t be able to fix fastcount for &lt;tt&gt;config.transactions&lt;/tt&gt;.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="565188">SERVER-35872</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="729836">SERVER-40482</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="757121">SERVER-40973</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="555530">SERVER-35483</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="720836">SERVER-40269</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="212027" name="txn-rollback.png" size="58477" author="max.hirschhorn@mongodb.com" created="Thu, 21 Mar 2019 23:01:24 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 25 Feb 2019 14:50:12 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 46 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/SERVER-35872'>SERVER-35872</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-1032</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 46 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>judah.schvimer@mongodb.com</customfieldvalue>
            <customfieldvalue>louis.williams@mongodb.com</customfieldvalue>
            <customfieldvalue>max.hirschhorn@mongodb.com</customfieldvalue>
            <customfieldvalue>pavithra.vetriselvan@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|huofrb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr7htb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2828">Storage NYC 2019-03-25</customfieldvalue>
    <customfieldvalue id="2829">Storage NYC 2019-04-08</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|huo20n:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>