<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:58:36 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-41766] Secondary may encounter prepare conflict when applying write that sets the multikey flag</title>
                <link>https://jira.mongodb.org/browse/SERVER-41766</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;There is a divergence around how we deal with setting the multikey flag on primary versus secondary. On primary, we go through this &lt;a href=&quot;https://github.com/mongodb/mongo/blob/48fa605e57d1a8d6d85f2a4f34b2c1af00052d69/src/mongo/db/catalog/index_catalog_entry_impl.cpp#L262-L288&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;codepath&lt;/a&gt;, and we register an onCommit handler on the OperationContext to switch the in-memory &lt;a href=&quot;https://github.com/mongodb/mongo/blob/7d7ed50c157e0665484a1142752bde6f676b7f61/src/mongo/db/catalog/index_catalog_entry_impl.h#L226-L228&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;multikey bit&lt;/a&gt; to true, which means we won&apos;t set it until the prepared transaction commits. So, that &lt;a href=&quot;https://github.com/mongodb/mongo/blob/48fa605e57d1a8d6d85f2a4f34b2c1af00052d69/src/mongo/db/catalog/index_catalog_entry_impl.cpp#L201-L205&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;allows&lt;/a&gt; subsequent writes to try updating the multikey flag again, since they see that the bit is not set. These writes would hit a prepare conflict on the collection catalog entry document and block, which is OK. On secondary, however, when we apply a prepared transaction, we appear to set the in-memory multikey bit immediately, instead of waiting for the prepared transaction to commit, since the OperationContext we use &lt;a href=&quot;https://github.com/mongodb/mongo/blob/dc9a28bff8ee8e20f764839f52172c458b24dbac/src/mongo/db/repl/sync_tail.cpp#L1391-L1393&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt; doesn&apos;t appear to be correlated to the actual transaction we applied and the WUOW &lt;a href=&quot;https://github.com/mongodb/mongo/blob/c36f9ecb91e49da7e637863889804fc4e6c6c05e/src/mongo/db/repl/storage_interface_impl.cpp#L550-L565&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;commits right away&lt;/a&gt;. Because of this behavior divergence, the following scenario, with two nodes, n0 and n1, is possible:&lt;/p&gt;

&lt;ol&gt;
	&lt;li&gt;A primary node (n0) prepares a transaction that did an insert that set the multikey flag. It does not update the in-memory multikey bit since the transaction has not committed yet.&lt;/li&gt;
	&lt;li&gt;The secondary (n1), applies the prepare operation that the primary just received. It &lt;em&gt;does&lt;/em&gt; set the in-memory multikey bit because of the behavior outlined above.&lt;/li&gt;
	&lt;li&gt;Node n1 steps up and becomes primary, n0 steps down and becomes secondary.&lt;/li&gt;
	&lt;li&gt;The new primary, n1, receives an insert operation that tries to set the multikey flag again. Because the in-memory bit has already been set, it does not try to write the flag again, and doesn&apos;t encounter a prepare conflict.&lt;/li&gt;
	&lt;li&gt;The current secondary, n0, tries to apply the insert operation that n1 just executed. Since n0 did not previously set the in-memory multikey bit, it tries to write the flag again, but it will hit a prepare conflict, since the transaction started in step (1) is still prepared. The secondary is now stuck in oplog application, unable to proceed.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;The attached repro ( &lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/220991/220991_multikey_prepare_txns.js&quot; title=&quot;multikey_prepare_txns.js attached to SERVER-41766&quot;&gt;multikey_prepare_txns.js&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; ) demonstrates this scenario. One possible suggestion to fix this issue is to do the multikey update in a side transaction on primary, so that it would not generate prepare conflicts at all. It should generally be safe to set the multikey flag on a collection earlier than is necessary, and it likely doesn&apos;t need to be atomic with respect to the original transaction that did the write. &lt;/p&gt;</description>
                <environment></environment>
        <key id="803621">SERVER-41766</key>
            <summary>Secondary may encounter prepare conflict when applying write that sets the multikey flag</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="william.schultz@mongodb.com">William Schultz</assignee>
                                    <reporter username="william.schultz@mongodb.com">William Schultz</reporter>
                        <labels>
                    </labels>
                <created>Fri, 14 Jun 2019 19:57:30 +0000</created>
                <updated>Sun, 29 Oct 2023 22:19:51 +0000</updated>
                            <resolved>Thu, 27 Jun 2019 18:24:38 +0000</resolved>
                                    <version>4.2.0-rc0</version>
                                    <fixVersion>4.2.0-rc3</fixVersion>
                    <fixVersion>4.3.1</fixVersion>
                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="2367655" author="xgen-internal-githook" created="Mon, 12 Aug 2019 20:25:25 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;William Schultz&apos;, &apos;email&apos;: &apos;william.schultz@mongodb.com&apos;, &apos;username&apos;: &apos;will62794&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-41766&quot; title=&quot;Secondary may encounter prepare conflict when applying write that sets the multikey flag&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-41766&quot;&gt;&lt;del&gt;SERVER-41766&lt;/del&gt;&lt;/a&gt; Remove obsolete code for tracking multikey writes inside multi-document transactions&lt;/p&gt;

&lt;p&gt;Now that we update the multikey flag within a transaction in a side transaction, the in-memory state about the multikey write will be naturally visible to subsequent writes inside the transaction, so we don&apos;t need to keep around any extra structures to enforce this anymore.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/20ba91db04c0b7b3d10fe2527b6938b1a14fcaa6&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/20ba91db04c0b7b3d10fe2527b6938b1a14fcaa6&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2309992" author="xgen-internal-githook" created="Tue, 2 Jul 2019 17:57:44 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;William Schultz&apos;, &apos;username&apos;: &apos;will62794&apos;, &apos;email&apos;: &apos;william.schultz@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-41766&quot; title=&quot;Secondary may encounter prepare conflict when applying write that sets the multikey flag&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-41766&quot;&gt;&lt;del&gt;SERVER-41766&lt;/del&gt;&lt;/a&gt; Update the on-disk multikey flag in a side transaction when running inside a multi-document transaction&lt;/p&gt;

&lt;p&gt;When a write inside a multi-document transaction needs to set an index as multikey, we update the multikey flag in the on-disk catalog in a transaction separate from the parent transaction. We commit this side transaction immediately, so as to avoid the catalog write generating prepare conflicts if it was written as part of a parent transaction that later became prepared. In general, it is safe to set an index as multikey too early. The multikey write is timestamped at the most recent value of the LogicalClock.&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 3147f5e1c37546b817934ef892d5e353170a9935)&lt;br/&gt;
Branch: v4.2&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/e549b451eb4ba9aa11f79a2356061f5dab6c943b&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/e549b451eb4ba9aa11f79a2356061f5dab6c943b&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2307045" author="william.schultz" created="Mon, 1 Jul 2019 16:05:05 +0000"  >&lt;p&gt;Additional notes from discussion on this ticket:&lt;/p&gt;

&lt;p&gt;Within a multi-document transaction, reads should be able to see the effect of previous writes done within that transaction. If a previous write in a transaction has set the index to be multikey, then a subsequent read must know that fact in order to return correct results. This is true in general for multikey writes. After this change, we update the multikey flag in memory and on-disk as soon as we do such a write inside a transaction. We do not wait for the transaction to commit. Thus, transactions should be able to read their own multikey writes correctly, without any &lt;a href=&quot;https://github.com/mongodb/mongo/blob/53d3a9e6d7b69787c0e4c3f23a930beef6f307bb/src/mongo/db/catalog/index_catalog_entry_impl.cpp#L321-L331&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;extra logic&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="2301119" author="xgen-internal-githook" created="Thu, 27 Jun 2019 18:24:21 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;William Schultz&apos;, &apos;email&apos;: &apos;william.schultz@mongodb.com&apos;, &apos;username&apos;: &apos;will62794&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-41766&quot; title=&quot;Secondary may encounter prepare conflict when applying write that sets the multikey flag&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-41766&quot;&gt;&lt;del&gt;SERVER-41766&lt;/del&gt;&lt;/a&gt; Update the on-disk multikey flag in a side transaction when running inside a multi-document transaction&lt;/p&gt;

&lt;p&gt;When a write inside a multi-document transaction needs to set an index as multikey, we update the multikey flag in the on-disk catalog in a transaction separate from the parent transaction. We commit this side transaction immediately, so as to avoid the catalog write generating prepare conflicts if it was written as part of a parent transaction that later became prepared. In general, it is safe to set an index as multikey too early. The multikey write is timestamped at the most recent value of the LogicalClock.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/3147f5e1c37546b817934ef892d5e353170a9935&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/3147f5e1c37546b817934ef892d5e353170a9935&lt;/a&gt;&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="537293">SERVER-34774</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="824787">SERVER-41988</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1157530">SERVER-46229</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1711537">SERVER-56877</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1870049">SERVER-59842</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="809731">SERVER-41848</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="220991" name="multikey_prepare_txns.js" size="1636" author="william.schultz@mongodb.com" created="Fri, 14 Jun 2019 19:57:52 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16775"><![CDATA[v4.2]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 27 Jun 2019 18:24:21 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 26 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 26 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>19.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>william.schultz@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hv5chz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hupc2v:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3001">Repl 2019-07-01</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hv4yrb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>