<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:08:11 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-25070] Significant delays when large collections are responsible for full WiredTiger cache</title>
                <link>https://jira.mongodb.org/browse/SERVER-25070</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We are unable to recreate this problem in 3.2.5. It appears to have existed since 3.2.7.&lt;/p&gt;

&lt;p&gt;One of our database systems which houses approximately 90 databases, most of which are less than 2GB in size, encounters an extreme degredation in performance when a particularly large collection (12GB) is loaded into the cache.  The cacheSizeGB on this system is 32GB. &lt;/p&gt;

&lt;p&gt;The problem only occurs if the WiredTiger cache is full AND the large collection appears loaded into the cache (speculation on the last part).  In this scenario, requests that typically take 1ms begin to take 50-500ms, and the slow request log blossoms quickly.&lt;/p&gt;

&lt;p&gt;We can replicate the above scenario by performing a simple query on the large collection when the WiredTiger cache is at or near its cache limit.&lt;/p&gt;

&lt;p&gt;We&apos;ve also been able to verify that the slowness does not happen when the database is otherwise taxed with a full cache. So we can execute a very long running map/reduce on other data and not impact performance. Only when the giant collections are loaded do we see the issue.&lt;/p&gt;</description>
                <environment></environment>
        <key id="301191">SERVER-25070</key>
            <summary>Significant delays when large collections are responsible for full WiredTiger cache</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="kelsey.schubert@mongodb.com">Kelsey Schubert</assignee>
                                    <reporter username="sallgeud">Chad Kreimendahl</reporter>
                        <labels>
                            <label>RF</label>
                    </labels>
                <created>Thu, 14 Jul 2016 18:28:07 +0000</created>
                <updated>Wed, 1 Feb 2017 00:45:47 +0000</updated>
                            <resolved>Mon, 30 Jan 2017 20:28:33 +0000</resolved>
                                    <version>3.2.7</version>
                    <version>3.2.8</version>
                                                    <component>Performance</component>
                    <component>WiredTiger</component>
                                        <votes>3</votes>
                                    <watches>19</watches>
                                                                                                                <comments>
                            <comment id="1489558" author="sallgeud" created="Tue, 31 Jan 2017 22:39:58 +0000"  >&lt;p&gt;That works. We&apos;ve not seen the problem since 3.2.10.&lt;/p&gt;</comment>
                            <comment id="1488439" author="thomas.schubert" created="Mon, 30 Jan 2017 20:28:23 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sallgeud&quot; class=&quot;user-hover&quot; rel=&quot;sallgeud&quot;&gt;sallgeud&lt;/a&gt;, &lt;/p&gt;

&lt;p&gt;We haven&#8217;t heard back from you for some time, so I&#8217;m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information and we will reopen the ticket.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1395077" author="thomas.schubert" created="Tue, 27 Sep 2016 20:51:27 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sallgeud&quot; class=&quot;user-hover&quot; rel=&quot;sallgeud&quot;&gt;sallgeud&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;I wrote a very detailed &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26055?focusedCommentId=1394968&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-1394968&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;comment&lt;/a&gt; on &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26055&quot; title=&quot;Server fills cache rapidly, even under replication only and then operates very slowing causing oplog backlog. When serving clients queries take ages to return once cache is &amp;#39;full&amp;#39;&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-26055&quot;&gt;&lt;del&gt;SERVER-26055&lt;/del&gt;&lt;/a&gt;, which describes the some of the fixes to eviction in WiredTiger that will be included in the upcoming release, MongoDB 3.2.10. &lt;/p&gt;

&lt;p&gt;In particular, I think this issue may be corrected by the third improvement that I noted. If this is the case, it is likely that it would also improve the behavior in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-25760&quot; title=&quot;Mongodump taking extraordinarily long, utilizing almost zero resources, yet slowing down server&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-25760&quot;&gt;&lt;del&gt;SERVER-25760&lt;/del&gt;&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-25663&quot; title=&quot;Odd connection timeouts and rejections when replicaset secondary is lagged&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-25663&quot;&gt;&lt;del&gt;SERVER-25663&lt;/del&gt;&lt;/a&gt;.  &lt;/p&gt;

&lt;p&gt;I would recommend upgrading to MongoDB 3.2.10 when it is released to confirm if this fixes the issue you are observing. Please note that mongoDB 3.2.10-rc2 was released today if you would like to test it, and we expect that MongoDB 3.2.10 will be released early next week.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Thomas &lt;/p&gt;</comment>
                            <comment id="1346297" author="sallgeud" created="Wed, 3 Aug 2016 19:10:40 +0000"  >&lt;p&gt;The workload basically stays the same. We can cause the problem by running that mapreduce on the massive collection (8GB+). The mapreduce is just an aggregation and validation of data, so it&apos;s only reads, no writes.&lt;/p&gt;</comment>
                            <comment id="1346143" author="thomas.schubert" created="Wed, 3 Aug 2016 16:21:42 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sallgeud&quot; class=&quot;user-hover&quot; rel=&quot;sallgeud&quot;&gt;sallgeud&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;In the last 2 hours preceding the restart (13:40), we&apos;re seeing a significant change the load. To help us continue to investigate this issue, would please provide additional details about your workload?&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;Can you clarify the change in the workload around this time?&lt;/li&gt;
	&lt;li&gt;In your description, you mention executing a long running map-reduce, is this when it starts? Does the map-reduce operation write to a collection?&lt;/li&gt;
	&lt;li&gt;Would you please provide the logs covering this time period? I have created a secure upload portal for you to use &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/4254f6ea-4025-462f-866e-cd90ab6f31c6.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Thank you,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1345556" author="sallgeud" created="Wed, 3 Aug 2016 03:57:50 +0000"  >&lt;p&gt;From our perspective the slow queries happen before the delay in the secondaries. What&apos;s strange to us is that the secondaries would fall behind at all when 90%+ of the commands running are queries and getmores. Are the secondaries getting lagged because the primary won&apos;t respond quickly enough? &lt;/p&gt;</comment>
                            <comment id="1345367" author="thomas.schubert" created="Tue, 2 Aug 2016 21:59:38 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sallgeud&quot; class=&quot;user-hover&quot; rel=&quot;sallgeud&quot;&gt;sallgeud&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;We are continuing to investigate this issue. However, I can provide some preliminary findings. We believe that &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-23622&quot; title=&quot;Inconsistent throughput during insert workload&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-23622&quot;&gt;&lt;del&gt;SERVER-23622&lt;/del&gt;&lt;/a&gt; is contributing to the behavior you are observing. There are two lagging replica set members, which causes the volume of the oplog in cache to grow. This is likely a positive feedback loop &#8211; cache pressure causes slow operations on the primary, which causes the secondaries tailing the oplog to lag, leading to more cache pressure on the primary and further slowness.&lt;/p&gt;

&lt;p&gt;We currently are examining whether there are other issues that may be impacting the performance you are seeing in addition to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-23622&quot; title=&quot;Inconsistent throughput during insert workload&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-23622&quot;&gt;&lt;del&gt;SERVER-23622&lt;/del&gt;&lt;/a&gt; and we will update you when we know more.&lt;/p&gt;

&lt;p&gt;Thank you for your help,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1345300" author="sallgeud" created="Tue, 2 Aug 2016 21:15:29 +0000"  >&lt;p&gt;Were you able to glean anything of value from the uploaded diagnostics?&lt;/p&gt;</comment>
                            <comment id="1324842" author="dan@10gen.com" created="Fri, 15 Jul 2016 02:25:52 +0000"  >&lt;p&gt;Hi Chad, thanks for the report.  Would you be able to upload the diagnostic.data for the impacted server?&lt;/p&gt;</comment>
                            <comment id="1324685" author="sallgeud" created="Thu, 14 Jul 2016 22:01:06 +0000"  >&lt;p&gt;This appears to relate to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-24775&quot; title=&quot;Degraded performance in 3.2.7 (probably after 3.2.5) - GlobalLock?&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-24775&quot;&gt;&lt;del&gt;SERVER-24775&lt;/del&gt;&lt;/a&gt; .&lt;/p&gt;

&lt;p&gt;Also, please replace my 3.2.6 with 3.2.5.  I verified that our system ran 3.2.5 without issues, but the upgrade to 3.2.7 is where they began.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="309727">SERVER-25663</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="311311">SERVER-25760</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="132960" name="During-DiagnosticData1.png" size="209194" author="kelsey.schubert@mongodb.com" created="Fri, 29 Jul 2016 05:17:12 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>10.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 15 Jul 2016 02:25:52 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        7 years, 2 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            7 years, 2 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>sallgeud</customfieldvalue>
            <customfieldvalue>dan@mongodb.com</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrk2lb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hsnkzj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[kelsey.schubert@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsa11b:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>