<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:23:46 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-30426] dropDatabase very slow due to repeated calls to closeAllCursors</title>
                <link>https://jira.mongodb.org/browse/SERVER-30426</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Example:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;dropDatabase with about 70 collections with 2 indexes each for a total of about 200 WT tables&lt;/li&gt;
	&lt;li&gt;about 325k open cursors&lt;/li&gt;
	&lt;li&gt;dropDatabase takes about 30 seconds and holds the global lock the entire time.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Collecting perf data shows mongod using 100% of a CPU for the duration, all in this stack:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;mongo::WiredTigerSession::closeAllCursors(std::__cxx11::basic_string&amp;lt;...&amp;gt; const&amp;amp;)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;mongo::WiredTigerSessionCache::closeAllCursors(std::__cxx11::basic_string&amp;lt;...&amp;gt; const&amp;amp;)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;mongo::WiredTigerKVEngine::_drop(mongo::StringData)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;mongo::WiredTigerKVEngine::dropIdent(mongo::OperationContext*, mongo::StringData)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;For each of the 200 WT tables dropped we call closeAllCursors for that table.  The perf stacks show all the time is spent in closeAllCursors itself, and the ftdc metrics also show that we aren&apos;t actually closing any cursors, so it appears that all the time is spent by closeAllCursors scanning the list of 325k open cursors to find cursors with a matching table uri.&lt;/p&gt;</description>
                <environment></environment>
        <key id="410391">SERVER-30426</key>
            <summary>dropDatabase very slow due to repeated calls to closeAllCursors</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="backlog-server-execution">Backlog - Storage Execution Team</assignee>
                                    <reporter username="bruce.lucas@mongodb.com">Bruce Lucas</reporter>
                        <labels>
                            <label>3.7BackgroundTask</label>
                    </labels>
                <created>Mon, 31 Jul 2017 13:34:24 +0000</created>
                <updated>Mon, 30 Oct 2023 23:14:45 +0000</updated>
                            <resolved>Mon, 21 May 2018 17:58:08 +0000</resolved>
                                    <version>3.2.15</version>
                    <version>3.4.6</version>
                                    <fixVersion>4.0.0-rc0</fixVersion>
                                    <component>Storage</component>
                                        <votes>29</votes>
                                    <watches>37</watches>
                                                                                                                <comments>
                            <comment id="1899976" author="milkie" created="Wed, 23 May 2018 19:45:51 +0000"  >&lt;p&gt;No, it takes effect immediately after the next cursor operation completes.&lt;/p&gt;</comment>
                            <comment id="1899967" author="mmillerick" created="Wed, 23 May 2018 19:38:45 +0000"  >&lt;p&gt;Does the mongo server need to be restarted in order for the new parameter value to take effect?&lt;/p&gt;</comment>
                            <comment id="1897339" author="milkie" created="Mon, 21 May 2018 17:58:08 +0000"  >&lt;p&gt;After &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33378&quot; title=&quot;Remove cursor caching from integration layer, using WiredTiger caching&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33378&quot;&gt;&lt;del&gt;SERVER-33378&lt;/del&gt;&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-32424&quot; title=&quot;Use WiredTiger cursor caching&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-32424&quot;&gt;&lt;del&gt;SERVER-32424&lt;/del&gt;&lt;/a&gt;, the number of cached cursors in each session is limited to the cursors that were used in the session since it was last checked out.  Our testing shows that this has greatly improved the performance of calls to WiredTigerSession::closeAllCursors().&lt;/p&gt;</comment>
                            <comment id="1897328" author="milkie" created="Mon, 21 May 2018 17:55:00 +0000"  >&lt;p&gt;In the linked server ticket &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33122&quot; title=&quot;add option to limit WiredTiger cursor cache size&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33122&quot;&gt;&lt;del&gt;SERVER-33122&lt;/del&gt;&lt;/a&gt;, you can find the parameter name: &quot;wiredTigerCursorCacheSize&quot;.  &lt;br/&gt;
In addition, we implemented a better cursor caching strategy in version 4.0, so I will be resolve this ticket now.&lt;/p&gt;</comment>
                            <comment id="1897258" author="mmillerick" created="Mon, 21 May 2018 17:14:34 +0000"  >&lt;blockquote&gt;
&lt;p&gt;We&apos;re actively investigating a temporary fix for this issue that would involve a server parameter to optionally turn off cursor caching.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;@milkie, what is the parameter?&lt;/p&gt;</comment>
                            <comment id="1708537" author="milkie" created="Wed, 25 Oct 2017 15:15:30 +0000"  >&lt;p&gt;I forgot to mention, the server parameter could be backported to 3.4, but the new drop behavior will not be.&lt;/p&gt;</comment>
                            <comment id="1708535" author="milkie" created="Wed, 25 Oct 2017 15:14:51 +0000"  >&lt;p&gt;We&apos;re actively investigating a temporary fix for this issue that would involve a server parameter to optionally turn off cursor caching.  This would result in a performance drop of up to 10%, but would completely avoid the drop issues and CPU consumption during collection and database drops.&lt;/p&gt;

&lt;p&gt;In addition, in version 3.6, we have enhanced the way dropDatabase works so that it effectively drops all collections in the database first, without holding the global lock, before dropping the now-empty database.  This won&apos;t affect the CPU consumption but it will improve server availability by significantly reducing the amount of time the global lock is held.&lt;/p&gt;</comment>
                            <comment id="1705730" author="alexander.gorrod" created="Mon, 23 Oct 2017 00:57:29 +0000"  >&lt;p&gt;The discussion in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-31101&quot; title=&quot;WT table not dropped after collection is dropped due to long-running OperationContext&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-31101&quot;&gt;&lt;del&gt;SERVER-31101&lt;/del&gt;&lt;/a&gt; is relevant to this ticket as well. It&apos;s likely that the fix for both issues will be related.&lt;/p&gt;</comment>
                            <comment id="1705376" author="aqueen" created="Fri, 20 Oct 2017 23:20:02 +0000"  >&lt;p&gt;I believe we&apos;re experiencing this on 3.4.9 as well. db.dropDatabase() on a roughly 2mb db takes minutes and essentially brings all dbs to a halt. Server has two processors, and we see close to 100% cpu on a single cpu where it just seems like something is spinning forever.&lt;/p&gt;

&lt;p&gt;I&apos;d like to know if this will be fixed soon and backported to 3.4 as it affects our operations when dropping databases. Is the proposed workaround for now to drop each collection individually, then the empty db afterward?&lt;/p&gt;</comment>
                            <comment id="1679427" author="david.henderson@triggeredmessaging.com" created="Fri, 22 Sep 2017 08:25:16 +0000"  >&lt;p&gt;Any progress on this issue - is it likely to hit a 3.4 release in the near future?&lt;/p&gt;</comment>
                            <comment id="1660652" author="bruce.lucas@10gen.com" created="Wed, 30 Aug 2017 14:46:29 +0000"  >&lt;p&gt;This problem was introduced by &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-27347&quot; title=&quot;Only close idle cached cursors on the WiredTiger ident that is busy&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-27347&quot;&gt;&lt;del&gt;SERVER-27347&lt;/del&gt;&lt;/a&gt;. Prior to that we would close all cached WT cursors on a failed table drop. The new behavior is better for a single table drop because it only closes relevant cursors, but may perform worse for a large number of failed table drops in a short period of time because it will repeatedly scan the list of cursors looking for the relevant cursors to close.&lt;/p&gt;</comment>
                            <comment id="1636285" author="richardyates" created="Mon, 31 Jul 2017 14:55:20 +0000"  >&lt;p&gt;Gotcha, thanks. I thought for second we had a serious open cursor issue!&lt;/p&gt;

&lt;p&gt;Let me/David know if we can be of further assistance with this one.&lt;/p&gt;

&lt;p&gt;&amp;#8211; Richard.&lt;/p&gt;</comment>
                            <comment id="1636275" author="bruce.lucas@10gen.com" created="Mon, 31 Jul 2017 14:43:03 +0000"  >&lt;p&gt;The 325k cursors here are internal: mongod maintains a cache of WT cursors to reduce the cost involved in creating and destroying WT cursors during normal operation. These cursors are WT cursors and are not the same thing as mongod client cursors, which is the number you are looking at. The size of that that cache grows to depends on factors like the number of open connections and the number of concurrent collections in use in the past, so it is expected that this number could be quite a bit larger than the number of mongodb client cursors currently open.&lt;/p&gt;</comment>
                            <comment id="1636249" author="richardyates" created="Mon, 31 Jul 2017 14:29:56 +0000"  >&lt;p&gt;Hi Bruce,&lt;/p&gt;

&lt;p&gt;A quick question if I may. The 325k open cursors, are those internal cursors or actual cursors from connections? We use mongodb cloud manager and it&apos;s saying the perticular server only has 2 open cursors so we&apos;re a little confused.&lt;/p&gt;

&lt;p&gt;&amp;#8211; Richard.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="492436">SERVER-33122</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                                        </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="406322">SERVER-30238</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="428152">SERVER-31101</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="337404">SERVER-27347</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>14.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25136"><![CDATA[Storage Execution]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[500A000000Z28RaIAJ, 500A000000ZgNOBIA3, 500A000000ZepS6IAJ, 500A000000bSiQUIA0, 500A000000bVANTIA4, 500A000000bWUUyIAO, 5002K00000hwXGcQAM]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 31 Jul 2017 14:29:56 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 38 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/SERVER-33122'>SERVER-33122</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>false</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 38 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>aqueen</customfieldvalue>
            <customfieldvalue>alexander.gorrod@mongodb.com</customfieldvalue>
            <customfieldvalue>backlog-server-execution</customfieldvalue>
            <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>david.henderson@triggeredmessaging.com</customfieldvalue>
            <customfieldvalue>milkie@mongodb.com</customfieldvalue>
            <customfieldvalue>mmillerick</customfieldvalue>
            <customfieldvalue>richardyates</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htbywf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr9cvj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htbkz3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>