<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:07:42 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-45037] CollectionBulkLoader::insertDocuments()  should be called with the collection cloner mutex held to guarantee memory and thread safety properties.</title>
                <link>https://jira.mongodb.org/browse/SERVER-45037</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;After this &lt;a href=&quot;https://github.com/mongodb/mongo/commit/a5af8f0d44f1cfef576241213039ee4730b7ed57&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;commit&lt;/a&gt;, &lt;a href=&quot;https://github.com/mongodb/mongo/blob/ca0c52357a2f92b37d4a3131199130507ad49bb8/src/mongo/db/repl/collection_cloner.cpp#L260&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;CollectionBulkLoader::insertDocuments() &lt;/a&gt; (storage document/ external sorter insertion) is not called with &lt;a href=&quot;https://github.com/mongodb/mongo/blob/ca0c52357a2f92b37d4a3131199130507ad49bb8/src/mongo/db/repl/collection_cloner.cpp#L248&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;collection cloner mutex&lt;/a&gt; held. As a result, we can hit data inconsistency and buffer overflow issues.&lt;/p&gt;

&lt;p&gt;Data inconsistency can occur because this CollectionBulkLoader::insertDocuments() is not thread-safe function.&lt;br/&gt;
 Buffer overflow can occur when the reads (returned from sync source) are much faster than the writes (document insertion). As a result, &lt;a href=&quot;https://github.com/mongodb/mongo/blob/ca0c52357a2f92b37d4a3131199130507ad49bb8/src/mongo/db/repl/collection_cloner.cpp#L210&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;CollectionCloner::_documentsToInsert&lt;/a&gt; buffer can overflow and lead to OOM issue.&lt;/p&gt;

&lt;p&gt;Note: I was able to see this buffer overflow, lead to server crash, when I tried to disable readOnce cursor on initial sync logkeeper workload.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1046476">SERVER-45037</key>
            <summary>CollectionBulkLoader::insertDocuments()  should be called with the collection cloner mutex held to guarantee memory and thread safety properties.</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="matthew.russotto@mongodb.com">Matthew Russotto</assignee>
                                    <reporter username="suganthi.mani@mongodb.com">Suganthi Mani</reporter>
                        <labels>
                    </labels>
                <created>Tue, 10 Dec 2019 15:46:42 +0000</created>
                <updated>Sun, 29 Oct 2023 22:14:16 +0000</updated>
                            <resolved>Thu, 2 Jan 2020 16:42:05 +0000</resolved>
                                                    <fixVersion>4.3.3</fixVersion>
                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>3</watches>
                                                                                                                <comments>
                            <comment id="2698792" author="xgen-internal-githook" created="Thu, 2 Jan 2020 16:34:45 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Matthew Russotto&apos;, &apos;email&apos;: &apos;matthew.russotto@mongodb.com&apos;, &apos;username&apos;: &apos;mtrussotto&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-45037&quot; title=&quot;CollectionBulkLoader::insertDocuments()  should be called with the collection cloner mutex held to guarantee memory and thread safety properties.&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-45037&quot;&gt;&lt;del&gt;SERVER-45037&lt;/del&gt;&lt;/a&gt; CollectionBulkLoader::insertDocuments()  should be called with the collection cloner mutex held to guarantee memory and thread safety properties.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/2c26afb53b1eefec26a09268c5c9e4fbe3b389d3&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/2c26afb53b1eefec26a09268c5c9e4fbe3b389d3&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2634416" author="matthew.russotto" created="Mon, 16 Dec 2019 16:44:22 +0000"  >&lt;p&gt;insertDocumentsCallback() is thread safe.  The crash is a result of there being no throttling, so the receive thread (handleNextBatch) is allowed to get far ahead of the insert thread, but this is not a race.  By inserting the documents while holding the cloner lock, we effectively force the receive thread to run in lock-step with insertion.   There may be a performance gain by using explicit throttling and not running in lock-step, but I think we just need to fix the bug for now.&lt;/p&gt;

&lt;p&gt;If the dbwork thread is being completely starved, we can still crash, but that case probably indicates some sort of system problem.&lt;/p&gt;</comment>
                            <comment id="2597134" author="suganthi.mani" created="Tue, 10 Dec 2019 18:54:53 +0000"  >&lt;p&gt;Thinking more about it, I feel, even after we revert to the old behavior (i.e) collectionBulkLoader::insertDocuments() should be called with the collection cloner mutex held, there is a slim chance that we can hit with the buffer overflow issue when your &lt;a href=&quot;https://github.com/mongodb/mongo/blob/ca0c52357a2f92b37d4a3131199130507ad49bb8/src/mongo/db/repl/collection_cloner.cpp#L215&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;task runner/thread pool&lt;/a&gt; is slow in scheduling the task. I feel we should have some limit to that CollectionCloner::_documentsToInsert buffer so that we cover the case of fast producer and slow consumer.&lt;/p&gt;</comment>
                            <comment id="2596698" author="suganthi.mani" created="Tue, 10 Dec 2019 16:17:38 +0000"  >&lt;p&gt;As a part of this ticket, I would like to add few other minor items&lt;/p&gt;

&lt;p&gt;1) correct the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/ca0c52357a2f92b37d4a3131199130507ad49bb8/src/mongo/db/repl/collection_cloner.cpp#L250&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;warning message&lt;/a&gt;-  s/_insertDocumentsCallback/insertDocumentsCallback&lt;br/&gt;
2) Add a note that CollectionBulkLoader::insertDocuments() is not thread safe to make it helpful for future readers.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10520">
                    <name>Problem/Incident</name>
                                                                <inwardlinks description="is caused by">
                                        <issuelink>
            <issuekey id="923548">SERVER-43274</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="2197992">SERVER-71683</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 16 Dec 2019 16:44:22 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 5 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 5 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>58.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>matthew.russotto@mongodb.com</customfieldvalue>
            <customfieldvalue>suganthi.mani@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hw9t07:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hvy0uf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3437">Repl 2019-12-16</customfieldvalue>
    <customfieldvalue id="3438">Repl 2019-12-30</customfieldvalue>
    <customfieldvalue id="3439">Repl 2020-01-13</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hw9f9j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>