<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:29:16 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-52879] Periodic operation latency spikes every 5 minutes due to closing idle cached WT sessions</title>
                <link>https://jira.mongodb.org/browse/SERVER-52879</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Test creates 1000 collections with 10 secondary indexes each (total 12 k tables) then does inserts at a rate of 1000 per second on 100 threads (10 per second for each thread) against randomly chosen collections. Latency is generally &amp;lt;1 ms but FTDC shows periodic latency spikes of tens of ms:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/287288/287288_spikes.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;The spikes coincide with calls to WiredTigerSessionCache::closeExpiredIdleSessions, and the interval between spikes is wiredTigerSessionCloseIdleTimeSecs, which has been set to 30 in the above test. I believe this is what&apos;s happening:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;closeExpiredIdleSessions takes a mutex on the WT session cache while closing idle sessions; this delays all operations.&lt;/li&gt;
	&lt;li&gt;Because of this up to 100 (number of client threads) operations queue up on the mutex and then proceed concurrently after closeExpiredIdleSessions completes, requiring up to 100 concurrent sessions to complete those operations; after completion those sessions are cached.&lt;/li&gt;
	&lt;li&gt;Between expiration intervals operations complete quickly so concurrency is low, so only a small number of the cached sessions are actually used.&lt;/li&gt;
	&lt;li&gt;So the next call to closeExpiredIdleSessions after the expiration interval finds that most of the cached sessions are idle, so it closes them while holding the session cache mutex, again delaying up to 100 operations and requiring them to open up to 100 new sessions after closeExpiredIdleSessions finishes closing the 100 old sessions, repeating the cycle.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;This causes the pattern of cached cursor count and open sessions that we see in the FTDC above; with the 1-second sample interval of FTDC we only occasionally catch the number of sessions decreasing due to closeExpiredIdleSessions before immediately increasing again, e.g. at A in the chart above.&lt;/p&gt;

&lt;p&gt;The magnitude of this effect increases with more tables; perf data shows that closeExpiredIdleSessions spends about 65% of its time in close_cached_cursors and about 25% in __session_discard_dhandle, both of which are proportional to the number of tables.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1543092">SERVER-52879</key>
            <summary>Periodic operation latency spikes every 5 minutes due to closing idle cached WT sessions</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="gregory.wlodarek@mongodb.com">Gregory Wlodarek</assignee>
                                    <reporter username="bruce.lucas@mongodb.com">Bruce Lucas</reporter>
                        <labels>
                    </labels>
                <created>Mon, 16 Nov 2020 15:15:10 +0000</created>
                <updated>Sun, 29 Oct 2023 22:00:22 +0000</updated>
                            <resolved>Thu, 7 Jan 2021 19:08:33 +0000</resolved>
                                    <version>4.4.1</version>
                    <version>4.2.10</version>
                    <version>4.0.21</version>
                    <version>3.6.21</version>
                                    <fixVersion>4.9.0</fixVersion>
                    <fixVersion>4.2.12</fixVersion>
                    <fixVersion>4.0.23</fixVersion>
                    <fixVersion>4.4.4</fixVersion>
                                                        <votes>0</votes>
                                    <watches>24</watches>
                                                                                                                <comments>
                            <comment id="3555301" author="xgen-internal-githook" created="Fri, 8 Jan 2021 17:45:28 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Gregory Wlodarek&apos;, &apos;email&apos;: &apos;gregory.wlodarek@mongodb.com&apos;, &apos;username&apos;: &apos;GWlodarek&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-52879&quot; title=&quot;Periodic operation latency spikes every 5 minutes due to closing idle cached WT sessions&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-52879&quot;&gt;&lt;del&gt;SERVER-52879&lt;/del&gt;&lt;/a&gt; Close expired idle sessions outside of the cache mutex&lt;/p&gt;

&lt;p&gt;(cherry picked from commit d517928d5b34e5988eea9fc15e1c2db4c3ff8d77)&lt;br/&gt;
Branch: v4.2&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/455bdbb8bcc2166a14fe009d28e71ee6bf4de59e&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/455bdbb8bcc2166a14fe009d28e71ee6bf4de59e&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3555290" author="xgen-internal-githook" created="Fri, 8 Jan 2021 17:42:23 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Gregory Wlodarek&apos;, &apos;email&apos;: &apos;gregory.wlodarek@mongodb.com&apos;, &apos;username&apos;: &apos;GWlodarek&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-52879&quot; title=&quot;Periodic operation latency spikes every 5 minutes due to closing idle cached WT sessions&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-52879&quot;&gt;&lt;del&gt;SERVER-52879&lt;/del&gt;&lt;/a&gt; Close expired idle sessions outside of the cache mutex&lt;/p&gt;

&lt;p&gt;(cherry picked from commit d517928d5b34e5988eea9fc15e1c2db4c3ff8d77)&lt;br/&gt;
Branch: v4.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/b01d1f1c14f1270e36adf06c78b6de9f63308936&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/b01d1f1c14f1270e36adf06c78b6de9f63308936&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3555080" author="xgen-internal-githook" created="Fri, 8 Jan 2021 16:26:06 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Gregory Wlodarek&apos;, &apos;email&apos;: &apos;gregory.wlodarek@mongodb.com&apos;, &apos;username&apos;: &apos;GWlodarek&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-52879&quot; title=&quot;Periodic operation latency spikes every 5 minutes due to closing idle cached WT sessions&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-52879&quot;&gt;&lt;del&gt;SERVER-52879&lt;/del&gt;&lt;/a&gt; Close expired idle sessions outside of the cache mutex&lt;/p&gt;

&lt;p&gt;(cherry picked from commit d517928d5b34e5988eea9fc15e1c2db4c3ff8d77)&lt;br/&gt;
Branch: v4.4&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/d6a3d7b707df26dfdc64b1ffd7264fae2886ec50&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/d6a3d7b707df26dfdc64b1ffd7264fae2886ec50&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3554823" author="bruce.lucas@10gen.com" created="Fri, 8 Jan 2021 14:47:22 +0000"  >&lt;p&gt;For better discoverability, I&apos;ve updated the title of this ticket to include &quot;every 5 minutes&quot;, based on the default value of wiredTigerSessionCloseIdleTimeSecs, to better describe the visible symptoms.&lt;/p&gt;</comment>
                            <comment id="3553222" author="xgen-internal-githook" created="Thu, 7 Jan 2021 18:07:13 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Gregory Wlodarek&apos;, &apos;email&apos;: &apos;gregory.wlodarek@mongodb.com&apos;, &apos;username&apos;: &apos;GWlodarek&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-52879&quot; title=&quot;Periodic operation latency spikes every 5 minutes due to closing idle cached WT sessions&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-52879&quot;&gt;&lt;del&gt;SERVER-52879&lt;/del&gt;&lt;/a&gt; Close expired idle sessions outside of the cache mutex&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/d517928d5b34e5988eea9fc15e1c2db4c3ff8d77&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/d517928d5b34e5988eea9fc15e1c2db4c3ff8d77&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3492480" author="bruce.lucas@10gen.com" created="Mon, 16 Nov 2020 19:49:38 +0000"  >&lt;p&gt;Also it would be helpful to add a metric reporting the cumulative number of sessions from the session cache that are closed - that would have made it much easier to diagnose the original customer issue.&lt;/p&gt;</comment>
                            <comment id="3491722" author="bruce.lucas@10gen.com" created="Mon, 16 Nov 2020 15:30:01 +0000"  >&lt;p&gt;Would it be possible to identify and remove the idle sessions from the cache under mutex, and then close those sessions outside of the mutex?&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                                        </outwardlinks>
                                                                <inwardlinks description="is related to">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="294125" name="after.png" size="68961" author="gregory.wlodarek@mongodb.com" created="Thu, 7 Jan 2021 14:45:08 +0000"/>
                            <attachment id="294124" name="before.png" size="68598" author="gregory.wlodarek@mongodb.com" created="Thu, 7 Jan 2021 14:45:02 +0000"/>
                            <attachment id="287290" name="repro.js" size="1064" author="bruce.lucas@mongodb.com" created="Mon, 16 Nov 2020 15:15:46 +0000"/>
                            <attachment id="287288" name="spikes.png" size="124864" author="bruce.lucas@mongodb.com" created="Mon, 16 Nov 2020 15:13:57 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="18953"><![CDATA[v4.4]]></customfieldvalue>
    <customfieldvalue key="16775"><![CDATA[v4.2]]></customfieldvalue>
    <customfieldvalue key="15640"><![CDATA[v4.0]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[5002K00000qejKNQAY, 5002K00000qg8MkQAI, 5002K00000s0IVRQA2, 5002K00000sdUs6QAE, 5002K00000uEDRLQA4]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 20 Nov 2020 00:41:34 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 4 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 4 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>gregory.wlodarek@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hyhjvb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr4kpr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="4365">Execution Team 2021-01-11</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hyh64n:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>