<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:50:46 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-39021] Switch migrations to observe multi-statement transaction CRUD statements onCommit instead of onCRUD</title>
                <link>https://jira.mongodb.org/browse/SERVER-39021</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;h2&gt;&lt;a name=&quot;ProblemSummary&quot;&gt;&lt;/a&gt;Problem Summary&lt;/h2&gt;
&lt;p&gt;We are proposing removing a collection X-lock in the &lt;tt&gt;MigrationSourceManager&lt;/tt&gt;. This X-lock creates a barrier after which all active transactions on the collection have been committed. This barrier ensures that the migration&apos;s chunk cloner will not miss writes that started before the migration.&lt;/p&gt;

&lt;p&gt;Without this barrier, the cloner will miss transaction writes that start before a migration starts. These writes will not be registered under the &lt;tt&gt;LogOpForShardingHandler&lt;/tt&gt;.&lt;/p&gt;

&lt;h2&gt;&lt;a name=&quot;ProposedApproach&quot;&gt;&lt;/a&gt;Proposed Approach&lt;/h2&gt;
&lt;p&gt;Because of this inconsistency, we would like to switch how migrations handle observing CRUD operations in multi-document transactions.&lt;/p&gt;

&lt;p&gt;After the change in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-39017&quot; title=&quot;Allow prepared transaction statements to persist in-memory until commit&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-39017&quot;&gt;&lt;del&gt;SERVER-39017&lt;/del&gt;&lt;/a&gt;, we will be able to observe all transaction statements on commit.Accordingly, &lt;a href=&quot;https://github.com/mongodb/mongo/blob/fcfe516250e2172d2fe6c9dab006b13d9d8140d7/src/mongo/db/op_observer_impl.cpp#L975-L1007&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;on the op observer onCommit handler&lt;/a&gt;, we can pass these statements to a new method on the &lt;tt&gt;OpObserverShardingImpl&lt;/tt&gt; class. This new method is defined in the next paragraph.&lt;/p&gt;

&lt;p&gt;A new method &lt;tt&gt;shardObserveTransactionCommit&lt;/tt&gt; placed on the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/master/src/mongo/db/s/op_observer_sharding_impl.cpp&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;OpObserverShardingImpl class&lt;/a&gt; will do the following:&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;Verify shard version &lt;em&gt;(parity with other CRUD op observers)&lt;/em&gt;.&lt;/li&gt;
	&lt;li&gt;Lock the CollectionShardingRuntimeLock &lt;em&gt;(parity with other CRUD op observers)&lt;/em&gt;.&lt;/li&gt;
	&lt;li&gt;Verify the existence of a migration &lt;em&gt;(parity with other CRUD op observers)&lt;/em&gt;.&lt;/li&gt;
	&lt;li&gt;Iterate over transaction statements and apply each CRUD op individually.&lt;/li&gt;
	&lt;li&gt;Verify that the chunk has not yet moved if in transaction &lt;em&gt;(parity with other CRUD op observers)&lt;/em&gt;.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;The previously stated iteration process will do the following with a vector of &lt;tt&gt;ReplOperations&lt;/tt&gt;.&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Check if the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/78a8fa36593cc0c429ffe10e07ccc33f7ce81d27/src/mongo/db/repl/oplog_entry.h#L115-L119&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;operation is CRUD type&lt;/a&gt;.&lt;/li&gt;
	&lt;li&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/78a8fa36593cc0c429ffe10e07ccc33f7ce81d27/src/mongo/db/repl/oplog_entry.h#L131-L137&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Extract the document&lt;/a&gt; from the &lt;tt&gt;ReplOperation&lt;/tt&gt; object.&lt;/li&gt;
	&lt;li&gt;&lt;tt&gt;switch&lt;/tt&gt; based on insert/update/delete and call the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/78a8fa36593cc0c429ffe10e07ccc33f7ce81d27/src/mongo/db/s/migration_chunk_cloner_source.h#L125-L158&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;corresponding chunk cloner CRUD op observer&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;NOTE: We previously stated that we needed an iterator class to analyze the list of replOperations. However, the iteration process is actually very simple &amp;#8211; for now, I think an anonymous function in the same file as the &lt;tt&gt;OpObserverShardingImpl&lt;/tt&gt; class will be sufficient.&lt;/p&gt;

&lt;p&gt;As a result of the previously stated work, we will now be duplicating ops sent to the shard observer &amp;#8211; we will be observing every individual CRUD operation, as well as those same operations if they are also inside a multi statement transaction. We don&apos;t want the migration to see the same write twice (since these updates are idempotent, it wouldn&apos;t be &lt;em&gt;incorrect&lt;/em&gt;, just &lt;em&gt;unnecessary&lt;/em&gt;). We should only &lt;a href=&quot;https://github.com/mongodb/mongo/blob/78a8fa36593cc0c429ffe10e07ccc33f7ce81d27/src/mongo/db/s/op_observer_sharding_impl.cpp#L92-L94&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;send writes to the cloner &lt;/a&gt; if we are not in a multi statement transaction, and a simple check of an already-given boolean will do that.&lt;/p&gt;
</description>
                <environment></environment>
        <key id="672920">SERVER-39021</key>
            <summary>Switch migrations to observe multi-statement transaction CRUD statements onCommit instead of onCRUD</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="blake.oler@mongodb.com">Blake Oler</assignee>
                                    <reporter username="blake.oler@mongodb.com">Blake Oler</reporter>
                        <labels>
                    </labels>
                <created>Tue, 15 Jan 2019 20:42:19 +0000</created>
                <updated>Sun, 29 Oct 2023 22:25:05 +0000</updated>
                            <resolved>Thu, 21 Feb 2019 21:10:29 +0000</resolved>
                                    <version>4.1.6</version>
                                    <fixVersion>4.1.9</fixVersion>
                                    <component>Sharding</component>
                                        <votes>0</votes>
                                    <watches>2</watches>
                                                                                                                <comments>
                            <comment id="2159042" author="xgen-internal-githook" created="Thu, 21 Feb 2019 21:07:38 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Blake Oler&apos;, &apos;email&apos;: &apos;blake.oler@mongodb.com&apos;, &apos;username&apos;: &apos;BlakeIsBlake&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-39021&quot; title=&quot;Switch migrations to observe multi-statement transaction CRUD statements onCommit instead of onCRUD&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-39021&quot;&gt;&lt;del&gt;SERVER-39021&lt;/del&gt;&lt;/a&gt; Switch migrations to observe multi-statement transaction CRUD statements onCommit instead of onCRUD&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/42999ae47bacedf260be2d6dd5e377ff0ae63744&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/42999ae47bacedf260be2d6dd5e377ff0ae63744&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2140373" author="pavithra.vetriselvan" created="Wed, 6 Feb 2019 21:49:22 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=blake.oler&quot; class=&quot;user-hover&quot; rel=&quot;blake.oler&quot;&gt;blake.oler&lt;/a&gt; Yes, feel free to CC me on this review!&lt;/p&gt;</comment>
                            <comment id="2140334" author="blake.oler" created="Wed, 6 Feb 2019 21:11:17 +0000"  >&lt;p&gt;As part of this ticket, we&apos;d like to change prepared transactions from:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Committing to storage&lt;/li&gt;
	&lt;li&gt;Observing transaction commit&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;to&lt;/p&gt;

&lt;ol&gt;
	&lt;li&gt;Observing transaction commit&lt;/li&gt;
	&lt;li&gt;Committing to storage&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;So that we can guarantee that we will be in a WriteUnitOfWork during observing the transaction commit. This will match the contract for op observers that observing always happens before storage commits. &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=pavithra.vetriselvan&quot; class=&quot;user-hover&quot; rel=&quot;pavithra.vetriselvan&quot;&gt;pavithra.vetriselvan&lt;/a&gt; gave the okay to change this ordering.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=pavithra.vetriselvan%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;pavithra.vetriselvan@mongodb.com&quot;&gt;pavithra.vetriselvan@mongodb.com&lt;/a&gt; shall I add you as a reviewer to my change for this ticket?&lt;/p&gt;</comment>
                            <comment id="2116312" author="renctan" created="Tue, 15 Jan 2019 21:27:05 +0000"  >&lt;p&gt;lgtm&lt;/p&gt;</comment>
                            <comment id="2116231" author="blake.oler" created="Tue, 15 Jan 2019 20:45:19 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kaloian.manassiev&quot; class=&quot;user-hover&quot; rel=&quot;kaloian.manassiev&quot;&gt;kaloian.manassiev&lt;/a&gt; or &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=renctan&quot; class=&quot;user-hover&quot; rel=&quot;renctan&quot;&gt;renctan&lt;/a&gt;, could I get an LGTM on this new approach?&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="672882">SERVER-39017</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is depended on by">
                                        <issuelink>
            <issuekey id="639246">SERVER-38284</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="665077">SERVER-38828</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 15 Jan 2019 21:27:05 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 50 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/SERVER-39017'>SERVER-39017</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-1286</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 50 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>blake.oler@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>pavithra.vetriselvan@mongodb.com</customfieldvalue>
            <customfieldvalue>randolph@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hujegv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hu43sf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2726">Sharding 2019-01-28</customfieldvalue>
    <customfieldvalue id="2786">Sharding 2019-02-11</customfieldvalue>
    <customfieldvalue id="2787">Sharding 2019-02-25</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|huj0q7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>