<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:19:43 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-29024] Collection drop slows/stalls down database operations</title>
                <link>https://jira.mongodb.org/browse/SERVER-29024</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Dropping collection that is &quot;active&quot; (I&apos;m guessing that used means not checkpointed), possibly if it&apos;s concurrent with an active checkpoint, leads to a visible slowdown (queries that took tens of milliseconds start taking hundreds an thousands of milliseconds) and bumps up the checkpoint time. We were hit by this since we upgraded to 3.4, and while the upgrade to 3.4.4 (&lt;a href=&quot;https://jira.mongodb.org/browse/WT-3207&quot; title=&quot;Drops with checkpoint_wait=false should not wait for checkpoints&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3207&quot;&gt;&lt;del&gt;WT-3207&lt;/del&gt;&lt;/a&gt;) helped with &quot;inactive&quot; collections (we create/drop collections that are grouped per-day), but dropping &quot;active&quot; collections still leads to latency spikes, and since the system is sized to latency N, when latency becomes 100*N it just can&apos;t cope with the load.&lt;br/&gt;
The worst part of it that entire database instance is affected, not just the collection or database in question.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-22199&quot; title=&quot;Collection drop command during checkpoint causes complete stall until end of checkpoint&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-22199&quot;&gt;&lt;del&gt;SERVER-22199&lt;/del&gt;&lt;/a&gt; + &lt;a href=&quot;https://jira.mongodb.org/browse/WT-2333&quot; title=&quot;Add a flag so drop doesn&amp;#39;t block&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-2333&quot;&gt;&lt;del&gt;WT-2333&lt;/del&gt;&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/browse/WT-2334&quot; title=&quot;Add support for background drops&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-2334&quot;&gt;&lt;del&gt;WT-2334&lt;/del&gt;&lt;/a&gt; seem to be related. &lt;/p&gt;

&lt;p&gt;Attaching the subject log. The WT checkpoint time which is 3.5 sec average for an hour before/after the event spiked to 25 sec (we&apos;ve seen 45 sec checkpoints in previous days).&lt;/p&gt;</description>
                <environment>Single-socket physical machine&lt;br/&gt;
XFS on a SATA SSD&lt;br/&gt;
Ubuntu 14.04 x64, 4.4 (HWE) kernel&lt;br/&gt;
MongoDB 3.4.4&lt;br/&gt;
</environment>
        <key id="379051">SERVER-29024</key>
            <summary>Collection drop slows/stalls down database operations</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="alexander.gorrod@mongodb.com">Alexander Gorrod</assignee>
                                    <reporter username="onyxmaster">Aristarkh Zagorodnikov</reporter>
                        <labels>
                    </labels>
                <created>Sun, 30 Apr 2017 14:07:48 +0000</created>
                <updated>Wed, 21 Jun 2017 21:23:27 +0000</updated>
                            <resolved>Tue, 23 May 2017 18:44:42 +0000</resolved>
                                    <version>3.4.4</version>
                                                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>10</watches>
                                                                                                                <comments>
                            <comment id="1578071" author="alexander.gorrod" created="Tue, 23 May 2017 18:44:01 +0000"  >&lt;blockquote&gt;&lt;p&gt;This time even drop of an inactive collection took &amp;gt;7 seconds, stalling other queries to different collections/databases for ~5 seconds. Please take a look, because this doesn&apos;t look like a cursor problem to me, we usually don&apos;t have long-running queries for this instance.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;It seems as though we haven&apos;t done a good job of explaining why some drops cause a temporary slowdown across all databases. The reason is that there is a cache of references to collections that is used to avoid the overhead of opening and closing those references for each operation. When a drop happens that cache becomes invalidated, and it can take a long time to re-populate the cache. The work in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-27347&quot; title=&quot;Only close idle cached cursors on the WiredTiger ident that is busy&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-27347&quot;&gt;&lt;del&gt;SERVER-27347&lt;/del&gt;&lt;/a&gt; aims to mitigate the cache invalidation when a drop is done - which should also mitigate the symptoms you are reporting.&lt;/p&gt;</comment>
                            <comment id="1564227" author="onyxmaster" created="Thu, 4 May 2017 07:31:33 +0000"  >&lt;p&gt;Added more logs and diagnostic.data from another instance. This time even drop of an inactive collection (at 2017-05-04T03:01:55.739+0300)  took &amp;gt;7 seconds, stalling other queries to different collections/databases for ~5 seconds. Please take a look, because this doesn&apos;t look like a cursor problem to me, we usually don&apos;t have long-running queries for this instance.&lt;/p&gt;</comment>
                            <comment id="1561716" author="alexander.gorrod" created="Mon, 1 May 2017 19:31:13 +0000"  >&lt;p&gt;It looks to me as though this is an instance where &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-27347&quot; title=&quot;Only close idle cached cursors on the WiredTiger ident that is busy&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-27347&quot;&gt;&lt;del&gt;SERVER-27347&lt;/del&gt;&lt;/a&gt; would help. The drop call causes MongoDB to flush the cursor cache, operations that follow shortly after the drop need to pay the cost of opening new cursors, rather than retrieving one from the cache. That causes performance to slow down, and the application consequently has more operations running in parallel than normal which require more cursors to be opened - since each connection uses it&apos;s own cursor. The particular statistic that indicates this problem is: &lt;tt&gt;ss wt session open cursor count&lt;/tt&gt; you can see that it drops very low at the same time as the drop, then grows again after the drop.&lt;/p&gt;

&lt;p&gt;I believe the longer checkpoint time is a consequence of the cursor cache flush/repopulate.&lt;/p&gt;

&lt;p&gt;Pictures below from the two instances I saw in the diagnostic data:&lt;/p&gt;

&lt;p&gt;This is the shorter of the two drop outs - there was also a change in workload prior to the drop that means there was additional load on the server.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/155015/155015_Screen+Shot+2017-05-01+at+3.26.51+pm.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;and&lt;/p&gt;

&lt;p&gt;This is the longer of the two drop outs, and it can be seen that re-opening the cursors takes quite a long time after the cache is flushed.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/155014/155014_Screen+Shot+2017-05-01+at+3.27.47+pm.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;</comment>
                            <comment id="1561201" author="onyxmaster" created="Mon, 1 May 2017 04:52:09 +0000"  >&lt;p&gt;Did cut out some less recent metrics to fit the 150MB limt, but I believe it would be enough.&lt;/p&gt;</comment>
                            <comment id="1561185" author="thomas.schubert" created="Mon, 1 May 2017 02:39:29 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=onyxmaster&quot; class=&quot;user-hover&quot; rel=&quot;onyxmaster&quot;&gt;onyxmaster&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for reporting this issue. To help us investigate, would you please upload an archive of the &lt;tt&gt;diagnostic.data&lt;/tt&gt; of the affected mongod?&lt;/p&gt;

&lt;p&gt;Thanks again,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="337404">SERVER-27347</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="155015" name="Screen Shot 2017-05-01 at 3.26.51 pm.png" size="72796" author="alexander.gorrod@mongodb.com" created="Mon, 1 May 2017 19:29:05 +0000"/>
                            <attachment id="155014" name="Screen Shot 2017-05-01 at 3.27.47 pm.png" size="73457" author="alexander.gorrod@mongodb.com" created="Mon, 1 May 2017 19:29:14 +0000"/>
                            <attachment id="155323" name="diagnostic.data-db5.tar" size="25589760" author="onyxmaster" created="Thu, 4 May 2017 07:28:09 +0000"/>
                            <attachment id="154958" name="diagnostic.data.tar" size="116075520" author="onyxmaster" created="Mon, 1 May 2017 04:51:36 +0000"/>
                            <attachment id="154959" name="event-diagnostics.png" size="366498" author="kelsey.schubert@mongodb.com" created="Mon, 1 May 2017 06:58:03 +0000"/>
                            <attachment id="154927" name="mongodb-db2.log.gz" size="1060018" author="onyxmaster" created="Sun, 30 Apr 2017 13:54:53 +0000"/>
                            <attachment id="155324" name="mongodb-drive2-db5.log.gz" size="124160" author="onyxmaster" created="Thu, 4 May 2017 07:27:35 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 1 May 2017 02:39:29 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        6 years, 38 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>backlog-server-pm</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            6 years, 38 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>alexander.gorrod@mongodb.com</customfieldvalue>
            <customfieldvalue>onyxmaster</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht6ntj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hsyvxj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hri2r3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>