<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:57:59 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-41529] To prevent dangling index records, CollectionBulkLoaderImpl should not call _addDocumentToIndexBlocks in a writeConflictRetry block. </title>
                <link>https://jira.mongodb.org/browse/SERVER-41529</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;For uncapped collections, CollectionBulkLoaderImpl::insertDocuments inserts documents by calling &lt;a href=&quot;https://github.com/mongodb/mongo/blob/452c7f88eb3aa4da8637b875006f82e04d496865/src/mongo/db/repl/collection_bulk_loader_impl.cpp#L133&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;CollectionImpl::insertDocumentForBulkLoader&lt;/a&gt; in a writeConflictRetry block. Regardless of this new batching design approach, it can lead to dangling index record entries. Consider, insertDocumentForBulkLoader() throws WriteConflictException. This leads to retrying of below operations.&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Record insertion&lt;/li&gt;
	&lt;li&gt;Inserting &amp;lt;Index key, RecordId&amp;gt; pair into external sorter.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Since inserting &amp;lt;Index key, RecordId&amp;gt; pair into external sorter is not part of storage transaction, previously failed storage transaction attempt would leave dangling index entries pointing to invalid RecordIds.&lt;/p&gt;

&lt;p&gt;The &lt;b&gt;solution&lt;/b&gt; to fix this bug is that we should not wrap the &#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/452c7f88eb3aa4da8637b875006f82e04d496865/src/mongo/db/repl/collection_bulk_loader_impl.cpp#L131&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;_addDocumentToIndexBlocks&lt;/a&gt;() method in a writeConflictRetry block.And, _addDocumentToIndexBlocks (insertion into external sorter) should be called only after &lt;a href=&quot;https://github.com/mongodb/mongo/blob/452c7f88eb3aa4da8637b875006f82e04d496865/src/mongo/db/repl/collection_bulk_loader_impl.cpp#L133&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;the batch of records&lt;/a&gt; got successfully committed in the storage&lt;/p&gt;</description>
                <environment></environment>
        <key id="789568">SERVER-41529</key>
            <summary>To prevent dangling index records, CollectionBulkLoaderImpl should not call _addDocumentToIndexBlocks in a writeConflictRetry block. </summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="allison.easton@mongodb.com">Allison Easton</assignee>
                                    <reporter username="suganthi.mani@mongodb.com">Suganthi Mani</reporter>
                        <labels>
                    </labels>
                <created>Wed, 5 Jun 2019 00:22:48 +0000</created>
                <updated>Sun, 29 Oct 2023 22:20:19 +0000</updated>
                            <resolved>Tue, 25 Jun 2019 21:26:19 +0000</resolved>
                                                    <fixVersion>4.2.0-rc2</fixVersion>
                    <fixVersion>4.3.1</fixVersion>
                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>5</watches>
                                                                                                                <comments>
                            <comment id="2520681" author="xgen-internal-githook" created="Wed, 6 Nov 2019 17:44:47 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Suganthi Mani&apos;, &apos;username&apos;: &apos;smani87&apos;, &apos;email&apos;: &apos;suganthi.mani@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-41529&quot; title=&quot;To prevent dangling index records, CollectionBulkLoaderImpl should not call _addDocumentToIndexBlocks in a writeConflictRetry block. &quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-41529&quot;&gt;&lt;del&gt;SERVER-41529&lt;/del&gt;&lt;/a&gt; Prevent dangling index records by calling _addDocumentToIndexBlock outside of writeConflictRetry block.&lt;/p&gt;

&lt;p&gt;(cherry picked from commit fc4c06660da1e121c817add86a56bbee1ef05f16)&lt;br/&gt;
(cherry picked from commit 54ca8a7112746c7637a295b6d57b6f2c3b4df9b7)&lt;br/&gt;
Branch: v4.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/d73429c822fd768ebb4ba4da93d108a9dc8536ad&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/d73429c822fd768ebb4ba4da93d108a9dc8536ad&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2311839" author="xgen-internal-githook" created="Wed, 3 Jul 2019 21:39:38 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Allison Easton&apos;, &apos;email&apos;: &apos;allison.easton@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-41529&quot; title=&quot;To prevent dangling index records, CollectionBulkLoaderImpl should not call _addDocumentToIndexBlocks in a writeConflictRetry block. &quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-41529&quot;&gt;&lt;del&gt;SERVER-41529&lt;/del&gt;&lt;/a&gt; Prevent dangling index records by calling&lt;br/&gt;
_addDocumentToIndexBlock outside of writeConflictRetry&lt;br/&gt;
block.&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 54ca8a7112746c7637a295b6d57b6f2c3b4df9b7)&lt;br/&gt;
Branch: v4.2&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/a54ca7e453bf0f291d5f1a825a0c92efab0b0b7b&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/a54ca7e453bf0f291d5f1a825a0c92efab0b0b7b&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2296388" author="xgen-internal-githook" created="Tue, 25 Jun 2019 13:12:52 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Allison Easton&apos;, &apos;email&apos;: &apos;allison.easton@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-41529&quot; title=&quot;To prevent dangling index records, CollectionBulkLoaderImpl should not call _addDocumentToIndexBlocks in a writeConflictRetry block. &quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-41529&quot;&gt;&lt;del&gt;SERVER-41529&lt;/del&gt;&lt;/a&gt; Prevent dangling index records by calling&lt;br/&gt;
_addDocumentToIndexBlock outside of writeConflictRetry&lt;br/&gt;
block.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/54ca8a7112746c7637a295b6d57b6f2c3b4df9b7&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/54ca8a7112746c7637a295b6d57b6f2c3b4df9b7&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2273275" author="suganthi.mani" created="Wed, 5 Jun 2019 22:20:49 +0000"  >&lt;p&gt;Contract is &lt;a href=&quot;https://github.com/mongodb/mongo/blob/452c7f88eb3aa4da8637b875006f82e04d496865/src/mongo/db/repl/collection_bulk_loader_impl.cpp#L131&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;_addDocumentToIndexBlocks()&lt;/a&gt; won&apos;t throw&#160;WriteConflictException and it&apos;s safe to assume that as it does only two things.&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;1) &lt;a href=&quot;https://github.com/mongodb/mongo/blob/452c7f88eb3aa4da8637b875006f82e04d496865/src/mongo/db/index/index_access_method.cpp#L587&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Extract&lt;/a&gt; the key values from the document.&lt;br/&gt;
 2) It would &lt;a href=&quot;https://github.com/mongodb/mongo/blob/452c7f88eb3aa4da8637b875006f82e04d496865/src/mongo/db/index/index_access_method.cpp#L604&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;insert&lt;/a&gt; the pair &amp;lt;Index key, RecordId&amp;gt; into an &lt;a href=&quot;https://github.com/mongodb/mongo/blob/452c7f88eb3aa4da8637b875006f82e04d496865/src/mongo/db/sorter/sorter.cpp#L500&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;in-memory buffer&lt;/a&gt; maintained by the external sorter.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;To be noted, we can&apos;t retry this step no:2 as that can cause dangling index entries while committing the indexes at the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/452c7f88eb3aa4da8637b875006f82e04d496865/src/mongo/db/repl/collection_cloner.cpp#L754&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;end of collection cloning&lt;/a&gt; . But&#160; my understanding is that this step &lt;a href=&quot;https://github.com/mongodb/mongo/blob/452c7f88eb3aa4da8637b875006f82e04d496865/src/mongo/db/catalog/collection_impl.cpp#L453&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;WiredTigerRecordStore::insertRecord&lt;/a&gt;&#160;can throw&#160;WriteConflictException.&#160;WriteConflictException is not just thrown for&#160;some conflicting writes and it can be thrown for other various reasons like wiredTiger memory issues. So, the solution I am suggesting is that we should wrap only the&#160;&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/452c7f88eb3aa4da8637b875006f82e04d496865/src/mongo/db/catalog/collection_impl.cpp#L453&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;WiredTigerRecordStore::insertRecord()&lt;/a&gt; in writeConflictRetry block and once the record insertion/writeUnitOfWork is successfully committed, then we should call addDocumentToIndexBlocks() with list of &amp;lt;doc,recordID&amp;gt; as we now know for sure the records are successfully inserted and committed and no way it can produce duplicate index record entries (same index keys with different recordIds.)&lt;/p&gt;</comment>
                            <comment id="2272346" author="judah.schvimer" created="Wed, 5 Jun 2019 14:54:05 +0000"  >&lt;p&gt;What should occur if it throws a &lt;tt&gt;WriteConflictException&lt;/tt&gt;? We do not want to have to restart the entire initial sync (though that&apos;s better than corrupted indexes).&lt;/p&gt;</comment>
                            <comment id="2271649" author="suganthi.mani" created="Wed, 5 Jun 2019 00:51:44 +0000"  >&lt;p&gt;As a part of this ticket, we should also write a js test to validate the fix.&lt;/p&gt;</comment>
                            <comment id="2271611" author="suganthi.mani" created="Wed, 5 Jun 2019 00:28:09 +0000"  >&lt;p&gt;This should be backported to 4.2 and 4.0.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="878848">SERVER-42484</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16775"><![CDATA[v4.2]]></customfieldvalue>
    <customfieldvalue key="15640"><![CDATA[v4.0]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 5 Jun 2019 14:54:05 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 14 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-1305</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 14 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>allison.easton@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>judah.schvimer@mongodb.com</customfieldvalue>
            <customfieldvalue>suganthi.mani@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hv32w7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|husacv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3000">Repl 2019-06-17</customfieldvalue>
    <customfieldvalue id="3001">Repl 2019-07-01</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hv2p5j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>