<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:10:51 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-26001] Insert workload stalled at 96% cache utilization</title>
                <link>https://jira.mongodb.org/browse/SERVER-26001</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Note: this may be the same underlying issue as &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-25974&quot; title=&quot;Application threads stall for extended period when cache fills&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-25974&quot;&gt;&lt;del&gt;SERVER-25974&lt;/del&gt;&lt;/a&gt;, but some of the metrics appear to be different, and this ticket has a specific simple repro not clearly tied to the customer issue seen on &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-25974&quot; title=&quot;Application threads stall for extended period when cache fills&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-25974&quot;&gt;&lt;del&gt;SERVER-25974&lt;/del&gt;&lt;/a&gt;, so opening as a separate ticket for now until/unless we can demonstrate that they are the same issue.&lt;/p&gt;

&lt;p&gt;The insert repro workload from &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-20306&quot; title=&quot;75% excess memory usage under WiredTiger during stress test&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-20306&quot;&gt;&lt;del&gt;SERVER-20306&lt;/del&gt;&lt;/a&gt;, also attached to this ticket as repro-32-insert.sh, gets stuck with cache utilization at about 96%:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/137760/137760_stuck.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;problems start at A, pretty much complete stuck at B&lt;/li&gt;
	&lt;li&gt;ftdc stalls (&quot;ftdc samples/s&quot;) suggest that application threads are sometimes getting stuck for extended periods doing evictions&lt;/li&gt;
	&lt;li&gt;application threads seem to be starved for work to do:
	&lt;ul&gt;
		&lt;li&gt;&quot;pages walked for eviction&quot; has gone up but &quot;pages seen by eviction walks&quot; has gone down&lt;/li&gt;
		&lt;li&gt;application threads are often finding the queue empty&lt;/li&gt;
		&lt;li&gt;pages evicted by application threads is not high&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;I&apos;ve also attached stack traces captured during the stuck period, although I don&apos;t think they give much insight.&lt;/p&gt;</description>
                <environment></environment>
        <key id="314673">SERVER-26001</key>
            <summary>Insert workload stalled at 96% cache utilization</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="david.hows">David Hows</assignee>
                                    <reporter username="bruce.lucas@mongodb.com">Bruce Lucas</reporter>
                        <labels>
                    </labels>
                <created>Wed, 7 Sep 2016 16:53:15 +0000</created>
                <updated>Wed, 26 Oct 2016 01:34:48 +0000</updated>
                            <resolved>Mon, 17 Oct 2016 12:44:08 +0000</resolved>
                                    <version>3.2.9</version>
                                                    <component>WiredTiger</component>
                                        <votes>2</votes>
                                    <watches>22</watches>
                                                                                                                <comments>
                            <comment id="1380160" author="david.hows" created="Fri, 9 Sep 2016 00:24:58 +0000"  >&lt;blockquote&gt;&lt;p&gt;My interpretation of the graph you uploaded is that the behavior is better with the more recent WiredTiger code.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Correct&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;It appears to me as though loading the first table finishes at around 3:35, where there is a small blip in insert operations. Loading the second table finishes at around 3:53. There is no evidence of dropouts during the run. &lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Correct&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;Wheras when run without the most recent version of WiredTiger the test was reliably failing to finish the load phase.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Correct&lt;/p&gt;</comment>
                            <comment id="1379291" author="alexander.gorrod" created="Thu, 8 Sep 2016 08:00:25 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=david.hows&quot; class=&quot;user-hover&quot; rel=&quot;david.hows&quot;&gt;david.hows&lt;/a&gt;. My interpretation of the graph you uploaded is that the behavior is better with the more recent WiredTiger code. It appears to me as though loading the first table finishes at around 3:35, where there is a small blip in insert operations. Loading the second table finishes at around 3:53. There is no evidence of dropouts during the run. Wheras when run &lt;b&gt;without&lt;/b&gt; the most recent version of WiredTiger the test was reliably failing to finish the load phase.&lt;/p&gt;

&lt;p&gt;Does that match your reading of the timeseries?&lt;/p&gt;</comment>
                            <comment id="1379290" author="david.hows" created="Thu, 8 Sep 2016 07:54:11 +0000"  >&lt;p&gt;I&apos;ve attached a screenshot of a timeseries of a run combining MongoDB 3.2.9 and WT Develop.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/137897/137897_david-screenshot.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;</comment>
                            <comment id="1379229" author="michael.cahill" created="Thu, 8 Sep 2016 06:20:50 +0000"  >&lt;p&gt;A little more detail here, I think what we&apos;re seeing here with 3.2.9 is a special case of what &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=alexander.gorrod&quot; class=&quot;user-hover&quot; rel=&quot;alexander.gorrod&quot;&gt;alexander.gorrod&lt;/a&gt; saw in &lt;a href=&quot;https://jira.mongodb.org/browse/WT-2894&quot; title=&quot;Create workload that shows negative scaling when overwhelmed&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-2894&quot;&gt;&lt;del&gt;WT-2894&lt;/del&gt;&lt;/a&gt; where eviction caused stalls in between checkpoints.  Because the configuration here disables checkpoints with &lt;tt&gt;--syncdelay 0&lt;/tt&gt;, if it gets into this state it never recovers, or takes an extremely long time.&lt;/p&gt;

&lt;p&gt;In particular, this is caused by a bug in the selection of eviction candidates where the eviction server will normally not retry eviction of a page if the transaction state hasn&apos;t changed since last time it tried.  This led to a busy wait where all active transactions were blocked waiting for eviction, and eviction was blocked waiting for something to change in the transaction state.&lt;/p&gt;

&lt;p&gt;This issue is resolved in WiredTiger&apos;s develop branch, and we plan to include those changes into 3.2.10.&lt;/p&gt;</comment>
                            <comment id="1379193" author="david.hows" created="Thu, 8 Sep 2016 04:01:25 +0000"  >&lt;p&gt;I&apos;ve run this and it reproduced readily for me on 3.2.9 pretty readily; during the first load stage (ping) I saw bulk inserts taking over 60 seconds to run rather than the normal few hundred milliseconds. This occurred at the 9.1GB inserted mark, which is also pretty indicative.&lt;/p&gt;

&lt;p&gt;I&apos;ve also made 3 full runs of this on v3.2 with the tip of WT develop, these did not reproduce the delays at all so I believe the issue is resolved within WT already.&lt;/p&gt;</comment>
                            <comment id="1378885" author="bruce.lucas@10gen.com" created="Wed, 7 Sep 2016 20:38:47 +0000"  >&lt;p&gt;That repro doesn&apos;t seem to be completely consistent. I&apos;ve run it a few times with a few variations on size and number of collections, and it only got completely stuck like that twice. The other times there would be significant stalls of maybe a minute or so, but it recovered and finished the run.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="314402">SERVER-25974</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="137897" name="david-screenshot.png" size="318964" author="david.hows" created="Thu, 8 Sep 2016 07:54:34 +0000"/>
                            <attachment id="137761" name="diagnostic.tar" size="511488" author="bruce.lucas@mongodb.com" created="Wed, 7 Sep 2016 16:53:15 +0000"/>
                            <attachment id="137768" name="repro-32-insert.sh" size="1372" author="bruce.lucas@mongodb.com" created="Wed, 7 Sep 2016 17:11:59 +0000"/>
                            <attachment id="137762" name="stacks1.txt" size="172504" author="bruce.lucas@mongodb.com" created="Wed, 7 Sep 2016 16:53:15 +0000"/>
                            <attachment id="137763" name="stacks2.txt" size="187204" author="bruce.lucas@mongodb.com" created="Wed, 7 Sep 2016 16:53:15 +0000"/>
                            <attachment id="137764" name="stacks3.txt" size="187204" author="bruce.lucas@mongodb.com" created="Wed, 7 Sep 2016 16:53:15 +0000"/>
                            <attachment id="137760" name="stuck.png" size="241167" author="bruce.lucas@mongodb.com" created="Wed, 7 Sep 2016 16:53:15 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 8 Sep 2016 04:01:25 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        7 years, 22 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>michael.cahill@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            7 years, 22 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>alexander.gorrod@mongodb.com</customfieldvalue>
            <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>david.hows</customfieldvalue>
            <customfieldvalue>michael.cahill@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrjwov:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hspimn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="1344">Storage 2016-10-31</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsd6f3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>