<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:31:13 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-53544] Verify prevOpTime chain for transactions is intact in resharding oplog fetcher</title>
                <link>https://jira.mongodb.org/browse/SERVER-53544</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/95c5037e67db28463465b34f33a62872226508eb/src/mongo/db/s/resharding/resharding_oplog_fetcher.cpp#L266&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;ReshardingOplogFetcher relies on {&lt;tt&gt;$_requestReshardingResumeToken: true&lt;/tt&gt;}&lt;/a&gt; &lt;a href=&quot;https://github.com/mongodb/mongo/blob/95c5037e67db28463465b34f33a62872226508eb/src/mongo/db/pipeline/pipeline_d.cpp#L683-L687&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;enabling QueryPlannerParams::ASSERT_MIN_TS_HAS_NOT_FALLEN_OFF_OPLOG&lt;/a&gt; and causing the aggregation pipeline to fail with an OplogQueryMinTsMissing error response if the donor&apos;s oplog rolls over. However, it is possible for the $graphLookup in the resharding oplog fetcher&apos;s aggregation pipeline to read from a later snapshot than the snapshot that was used for the collection scan to populate the DocumentSourceCursor. This means the following sequence could happen:&lt;/p&gt;

&lt;ol&gt;
	&lt;li&gt;ReshardingOplogFetcher runs [{$match: {ts: {$gte: Timestamp(10, 1)}}}, ..., {$graphLookup: ...}, ...].&lt;/li&gt;
	&lt;li&gt;CollectionScan verifies the {ts: Timestamp(10, 1)} document is still present in the oplog. The {ts: Timestamp(10, 1)} document has {prevOpTime: {ts: Timestamp(5, 1), t: 1}}.&lt;/li&gt;
	&lt;li&gt;DocumentSourceCursor buffers a batch of documents.&lt;/li&gt;
	&lt;li&gt;Oplog truncation occurs and now {ts: Timestamp(6, 1)} is the oldest oplog entry.&lt;/li&gt;
	&lt;li&gt;$graphLookup finds the {ts: Timestamp(10, 1)} document but fails to find a {ts: Timestamp(5, 1)} document.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;The ReshardingOplogFetcher could confirm the prevOpTime value is {prevOpTime: {ts: Timestamp(0, 0), t: -1}} whenever it sees a new transaction. The aggregation pipeline that resharding&apos;s oplog fetching is using guarantees all of the oplog entries from the same multi-document transaction are contiguous so it only needs to track this for a single multi-document transaction at a time. We may as well verify the entire prevOpTime chain values for the later entries too.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1578634">SERVER-53544</key>
            <summary>Verify prevOpTime chain for transactions is intact in resharding oplog fetcher</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="max.hirschhorn@mongodb.com">Max Hirschhorn</assignee>
                                    <reporter username="max.hirschhorn@mongodb.com">Max Hirschhorn</reporter>
                        <labels>
                            <label>PM-234-M3</label>
                            <label>PM-234-T-oplog-fetch</label>
                            <label>post-rc0</label>
                    </labels>
                <created>Wed, 30 Dec 2020 17:18:40 +0000</created>
                <updated>Sat, 5 Jun 2021 14:20:11 +0000</updated>
                            <resolved>Sat, 5 Jun 2021 14:20:06 +0000</resolved>
                                                                    <component>Sharding</component>
                                        <votes>0</votes>
                                    <watches>1</watches>
                                                                                                                <comments>
                            <comment id="3861234" author="max.hirschhorn@10gen.com" created="Sat, 5 Jun 2021 14:20:06 +0000"  >&lt;p&gt;This issue has been addressed by the changes from &lt;a href=&quot;https://github.com/mongodb/mongo/commit/b4c7db3eec365033a85c407e7b580ee5ed6518bd&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;b4c7db3&lt;/a&gt; as part of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-57303&quot; title=&quot;Create transaction history iterator stage for Resharding&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-57303&quot;&gt;&lt;del&gt;SERVER-57303&lt;/del&gt;&lt;/a&gt; by having the TransactionHistoryIterator propagate the IncompleteTransactionHistory exception when the prevOpTime chain isn&apos;t intact. Moreover, resharding&apos;s oplog fetching pipeline no longer uses $lookup or $graphLookup so the non&amp;#45;transaction oplog entries are not at risk of being elided by reading from different snapshots between stages within the aggregation pipeline.&lt;/p&gt;</comment>
                            <comment id="3544792" author="max.hirschhorn@10gen.com" created="Wed, 30 Dec 2020 17:22:26 +0000"  >&lt;p&gt;I think there&apos;s a related edge case where the $graphLookup and $lookup fail to find the {ts: Timestamp(10, 1)} document itself.&lt;/p&gt;

&lt;ol&gt;
	&lt;li&gt;ReshardingOplogFetcher runs [{$match: {ts: {$gte: Timestamp(10, 1)}}}, ..., {$graphLookup: ...}, ...].&lt;/li&gt;
	&lt;li&gt;CollectionScan verifies the {ts: Timestamp(10, 1)} document is still present in the oplog.&lt;/li&gt;
	&lt;li&gt;DocumentSourceCursor buffers a batch of documents.&lt;/li&gt;
	&lt;li&gt;Oplog truncation occurs and now {ts: Timestamp(11, 1)} is the oldest oplog entry.&lt;/li&gt;
	&lt;li&gt;$graphLookup fails to find a {ts: Timestamp(10, 1)} document.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;I suspect using {&lt;tt&gt;preserveNullAndEmptyArrays: true&lt;/tt&gt;} with the $unwind stages that follow the $graphLookup and the $lookup would pass through enough information that we &quot;lost&quot; an oplog entry from the donor shard.&lt;/p&gt;

&lt;p&gt;We could also consider a different approach to address both scenarios of having recipient shards prevent donor shards from truncation portions of their oplog. Or having donor shards attach the timestamp of their oldest oplog entry in the cursor response and doing the necessary plumbing in PlanExecutorPipeline (see &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-53534&quot; title=&quot;Support including postBatchResumeToken in cursor response for non-change streams aggregations&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-53534&quot;&gt;&lt;del&gt;SERVER-53534&lt;/del&gt;&lt;/a&gt;) so ReshardingOplogFetcher can error out itself.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="1762017">SERVER-57303</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="1421051">SERVER-49892</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="21777"><![CDATA[v5.0]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 35 weeks, 4 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-234</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>max.hirschhorn@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 35 weeks, 4 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>max.hirschhorn@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hynkkn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hy9d47:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="4519">Sharding 2021-06-14</customfieldvalue>
    <customfieldvalue id="4789">Sharding 2021-05-31</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10555" key="com.atlassian.jira.plugin.system.customfieldtypes:float">
                        <customfieldname>Story Points</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hyn6tz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>