<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:05:38 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-24194] Queued table drops within the WiredTiger KVEngine can compete with each other </title>
                <link>https://jira.mongodb.org/browse/SERVER-24194</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Queued table drops within the WTKVEngine can compete with each other.&lt;/p&gt;

&lt;p&gt;When creating and dropping a large number of tables (such as running multiple concurrent map/reduce jobs) MongoDB creates a large number of tables then drops them in quick succession. Given that the instance is still &quot;hot&quot; at this point as there are multiple create and drop operations going on at once, many of these drops get queued to be dropped later.&lt;/p&gt;

&lt;p&gt;This queuing then triggers a close of all sessions and any number of threads will then enter the code path to drop all of the queued collections all starting at the top. Having multiple threads enter the drop code at once means that we often see several attempts to drop one collection(s) before a single drop succeeds, during which time further collection drops are often queued. This causes an almost quadratic case, where a high rate of collection creates/drops will cause more and more drops to be queued.&lt;/p&gt;

&lt;p&gt;One thing needed to create the conditions for this is a small level of delay in processing of the deletes. I suspect this occurs on OSX as we must use the slower fsync or with strace on linux - which slows the OS calls.&lt;/p&gt;</description>
                <environment></environment>
        <key id="287532">SERVER-24194</key>
            <summary>Queued table drops within the WiredTiger KVEngine can compete with each other </summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="david.hows">David Hows</assignee>
                                    <reporter username="david.hows">David Hows</reporter>
                        <labels>
                            <label>code-only</label>
                    </labels>
                <created>Wed, 18 May 2016 14:37:35 +0000</created>
                <updated>Tue, 22 Nov 2016 14:14:50 +0000</updated>
                            <resolved>Wed, 1 Jun 2016 01:19:38 +0000</resolved>
                                                    <fixVersion>3.2.8</fixVersion>
                    <fixVersion>3.3.8</fixVersion>
                                    <component>Storage</component>
                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>6</watches>
                                                                                                                <comments>
                            <comment id="1305170" author="xgen-internal-githook" created="Fri, 24 Jun 2016 06:17:38 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;daveh86&apos;, u&apos;name&apos;: u&apos;David Hows&apos;, u&apos;email&apos;: u&apos;howsdav@gmail.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-24194&quot; title=&quot;Queued table drops within the WiredTiger KVEngine can compete with each other &quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-24194&quot;&gt;&lt;del&gt;SERVER-24194&lt;/del&gt;&lt;/a&gt; - Backport, change the dropAllQueued loop to avoid duplicating work over many threads&lt;br/&gt;
Branch: v3.2&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/ba5fe233b5915d535a7b73da911b6b9fe132c035&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/ba5fe233b5915d535a7b73da911b6b9fe132c035&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1280430" author="xgen-internal-githook" created="Wed, 1 Jun 2016 01:19:11 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;daveh86&apos;, u&apos;name&apos;: u&apos;David Hows&apos;, u&apos;email&apos;: u&apos;howsdav@gmail.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-24194&quot; title=&quot;Queued table drops within the WiredTiger KVEngine can compete with each other &quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-24194&quot;&gt;&lt;del&gt;SERVER-24194&lt;/del&gt;&lt;/a&gt; - switch from the WiredTiger lock_wait flag to the checkpoint_wait flag&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/8b13ccc77395e153a607e49049323d07d7b7f9da&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/8b13ccc77395e153a607e49049323d07d7b7f9da&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1275765" author="david.hows" created="Thu, 26 May 2016 01:51:00 +0000"  >&lt;p&gt;I&apos;ve made an initial commit here of the logical changes to the dropAllQueued function (now renamed dropSomeQueuedIdents). Will need one further change when the changes in &lt;a href=&quot;https://jira.mongodb.org/browse/WT-2646&quot; title=&quot;Split the lock_wait flag into two, adding a checkpoint_wait flag&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-2646&quot;&gt;&lt;del&gt;WT-2646&lt;/del&gt;&lt;/a&gt; are moved upstream into MongoDB.&lt;/p&gt;</comment>
                            <comment id="1275761" author="xgen-internal-githook" created="Thu, 26 May 2016 01:44:54 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;daveh86&apos;, u&apos;name&apos;: u&apos;David Hows&apos;, u&apos;email&apos;: u&apos;howsdav@gmail.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-24194&quot; title=&quot;Queued table drops within the WiredTiger KVEngine can compete with each other &quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-24194&quot;&gt;&lt;del&gt;SERVER-24194&lt;/del&gt;&lt;/a&gt; - Change the dropAllQueued loop to avoid duplicating work over many threads&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/cc6ee9cf116853289ee41784220b3ce8ed14c29c&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/cc6ee9cf116853289ee41784220b3ce8ed14c29c&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1268125" author="david.hows" created="Wed, 18 May 2016 20:30:58 +0000"  >&lt;p&gt;The modifications I made to remove locking overlap between new drop&apos;s being performed and the dropAllQueued operation running have made the original issue re-appear. Back to the drawing board.&lt;/p&gt;</comment>
                            <comment id="1268114" author="david.hows" created="Wed, 18 May 2016 20:23:53 +0000"  >&lt;p&gt;Good questions &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=agorrod&quot; class=&quot;user-hover&quot; rel=&quot;agorrod&quot;&gt;agorrod&lt;/a&gt;.&lt;/p&gt;

&lt;blockquote&gt;&lt;p&gt;Does this put a greater time burden on a single thread? &lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Not really. Currently all the threads are walking the whole queue anyway, now it is just one. That one may do slightly more work, as it may now get a chance to acquire the 3 WT locks involved in a drop and actually complete the table drop operation. Another thing of note is that the WT table drop call did not originally have the force and lock_wait=false flags set, these have been set more recently and should mean we never hit a heavyweight operation as part of the drop (that is left to other threads within WT).&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;Can dropAllQueued be called in an application thread? &lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Yes. From any thread that uses a WiredTigerSession object&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;If so, could it end up spending an unfair amount of time doing the drops?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Potentially possible. But I can&apos;t envision a scenario which is worse than the current behaviour, as each thread must walk the whole list of tables to drop anyway.&lt;/p&gt;

&lt;blockquote&gt;&lt;p&gt;Given that the _identToDropMutex is being held for longer in dropAllQueued - is it a concern that calls to WiredTigerKVEngine::_drop could need to wait for the lock?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;I&apos;ve updated the patch to avoid this problem. Thanks&lt;/p&gt;

&lt;blockquote&gt;&lt;p&gt;Are there tests we can run that will give us confidence there won&apos;t be negative consequences?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Not sure, but I will try and either find some or write a test case.&lt;/p&gt;</comment>
                            <comment id="1267999" author="alexander.gorrod" created="Wed, 18 May 2016 18:51:04 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=david.hows&quot; class=&quot;user-hover&quot; rel=&quot;david.hows&quot;&gt;david.hows&lt;/a&gt; Thanks - I&apos;m glad you got to the bottom of this failure. I&apos;m concerned that switching to a single threaded approach here will have negative consequences on some workloads. I can think of two potential degradations:&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;Does this put a greater time burden on a single thread? Can &lt;tt&gt;dropAllQueued&lt;/tt&gt; be called in an application thread? If so, could it end up spending an unfair amount of time doing the drops?&lt;/li&gt;
	&lt;li&gt;Given that the &lt;tt&gt;_identToDropMutex&lt;/tt&gt; is being held for longer in &lt;tt&gt;dropAllQueued&lt;/tt&gt; - is it a concern that calls to &lt;tt&gt;WiredTigerKVEngine::_drop&lt;/tt&gt; could need to wait for the lock?&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Are there tests we can run that will give us confidence there won&apos;t be negative consequences? &lt;/p&gt;</comment>
                            <comment id="1267666" author="david.hows" created="Wed, 18 May 2016 14:58:54 +0000"  >&lt;p&gt;Two things to consider here in terms of solution:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Single thread the dropAllQueues function to avoid quadratic build up&lt;/li&gt;
	&lt;li&gt;Look at logic to avoid queuing the cacheing of WT cursors on temp tables.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;The cacheing of cursors on temp tables mentioned above is one of the initial conditions that leads to the initial inability to drop a table. If we can remove some of the initial conditions, this will help avoid causing problems.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="287988">WT-2646</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12451" key="com.atlassian.jira.plugin.system.customfieldtypes:multiversion">
                        <customfieldname>Backport Completed</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="16982">3.2.8</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 18 May 2016 18:51:04 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        7 years, 33 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ian@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            7 years, 33 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>alexander.gorrod@mongodb.com</customfieldvalue>
            <customfieldvalue>david.hows</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrk7rr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hslk53:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;Run the test jstest/slow1/remove_during_mr.js on OSX or on Linux with strace instrumentation to slow system calls.&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrot8f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>