<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:03:57 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-23622] Inconsistent throughput during insert workload</title>
                <link>https://jira.mongodb.org/browse/SERVER-23622</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;With a heavy insert load and a secondary that is delayed the cache on the primary fills to 100% and operation rates drop. Here&apos;s a run showing behavior on the primary with the secondary delayed due to lag, but a similar effect is seen if the secondary is intentionally delayed using slaveDelay.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/117090/117090_lagging.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;from D-E the cache is 95% full and insert rate drops considerably, possibly due to application threads doing evictions?&lt;/li&gt;
	&lt;li&gt;F-G and H-I seem to be seem to be related to checkpoints, possibly also in combination with the full cache?&lt;/li&gt;
	&lt;li&gt;the rate of pages walked for eviction is generally very high, about 6k times the rate of pages actually evicted, suggesting that the issue is difficulty finding pages to evict to keep the cache at target levels&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;The high rate of pages walked for eviction suggests a connection to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-22831&quot; title=&quot;Low query rate with heavy cache pressure and an idle collection&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-22831&quot;&gt;&lt;del&gt;SERVER-22831&lt;/del&gt;&lt;/a&gt;, which also showed that symptom in connection with a full cache; however the above run was on 3.2.5-rc1 where &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-22831&quot; title=&quot;Low query rate with heavy cache pressure and an idle collection&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-22831&quot;&gt;&lt;del&gt;SERVER-22831&lt;/del&gt;&lt;/a&gt; was fixed, so it seems there is a different issue here.&lt;/p&gt;

&lt;p&gt;The above test involved&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;3.2.5-rc1&lt;/li&gt;
	&lt;li&gt;2-node replica set&lt;/li&gt;
	&lt;li&gt;25 GB cache&lt;/li&gt;
	&lt;li&gt;100 GB oplog&lt;/li&gt;
	&lt;li&gt;5 threads inserting 800 byte documents into 5 separate collections&lt;/li&gt;
&lt;/ul&gt;


&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;for t in $(seq 5); do&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    mongo --eval &quot;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        x = &apos;&apos;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        for (var i=0; i&amp;lt;800; i++)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            x += &apos;x&apos;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        docs = []&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        for (var i=0; i&amp;lt;1000; i++)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            docs.push({x:x})&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        ops = [{&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            op: &apos;insert&apos;,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            ns: &apos;test.c&apos; + &apos;$t&apos;,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            doc: docs,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        }]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        res = benchRun({&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            ops: ops,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            seconds: 10000,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            parallel: 1&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        })&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    &quot; &amp;amp;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;done&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;wait&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</description>
                <environment></environment>
        <key id="278525">SERVER-23622</key>
            <summary>Inconsistent throughput during insert workload</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="sue.loverso@mongodb.com">Susan LoVerso</assignee>
                                    <reporter username="bruce.lucas@mongodb.com">Bruce Lucas</reporter>
                        <labels>
                    </labels>
                <created>Fri, 8 Apr 2016 19:59:53 +0000</created>
                <updated>Wed, 21 Dec 2016 23:35:50 +0000</updated>
                            <resolved>Thu, 10 Nov 2016 22:00:08 +0000</resolved>
                                                                    <component>WiredTiger</component>
                                        <votes>1</votes>
                                    <watches>20</watches>
                                                                                                                <comments>
                            <comment id="1431510" author="michael.cahill" created="Thu, 10 Nov 2016 22:00:08 +0000"  >&lt;p&gt;Closing this issue because it ended up focusing on standalone performance.  Opened &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26988&quot; title=&quot;Secondary delay causes large drop in insert rate on the primary due to cache full condition&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-26988&quot;&gt;&lt;del&gt;SERVER-26988&lt;/del&gt;&lt;/a&gt; to track the original reported issue of cache pressure caused by a delayed secondary.&lt;/p&gt;</comment>
                            <comment id="1431501" author="sue.loverso" created="Thu, 10 Nov 2016 21:52:16 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=michael.cahill&quot; class=&quot;user-hover&quot; rel=&quot;michael.cahill&quot;&gt;michael.cahill&lt;/a&gt; Is there more to do in this ticket?  We have the changes from &lt;a href=&quot;https://jira.mongodb.org/browse/WT-2816&quot; title=&quot;Improve WiredTiger eviction performance&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-2816&quot;&gt;&lt;del&gt;WT-2816&lt;/del&gt;&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/browse/WT-2866&quot; title=&quot;Eviction server algorithm tuning&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-2866&quot;&gt;&lt;del&gt;WT-2866&lt;/del&gt;&lt;/a&gt;.  I do still see occasional long latencies when running, but not dropouts in the time before my disk fills.  When I ran on my old, very large SSD AWS box it ran for an hour with 3 drops to zero, all exactly 15 minutes apart.  I assume those were some kind of periodic SSD housekeeping event.&lt;/p&gt;
</comment>
                            <comment id="1379098" author="michael.cahill" created="Thu, 8 Sep 2016 00:18:54 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sue.loverso&quot; class=&quot;user-hover&quot; rel=&quot;sue.loverso&quot;&gt;sue.loverso&lt;/a&gt;, thanks for the new data.  I see two primary causes of the remaining stalls:&lt;/p&gt;

&lt;p&gt;1. threads contending to split a large page when appending: this leads to plateaus in &quot;range of IDs currently pinned&quot; causing a stall; and&lt;br/&gt;
2. ineffective scrubbing on the tip of the &lt;a href=&quot;https://jira.mongodb.org/browse/WT-2233&quot; title=&quot;Investigate changing when the eviction server switches to aggressive mode.&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-2233&quot;&gt;&lt;del&gt;WT-2233&lt;/del&gt;&lt;/a&gt; branch leading to stalls during checkpoints (because we&apos;re at the dirty limit and checkpoint blocks eviction of dirty pages).&lt;/p&gt;

&lt;p&gt;I&apos;ll fix 2 first up this morning: the issue there is that scrubbing was the cause of the regression in the YCSB 50/50 workload, so what we want is something that allows scrubbing to reduce the amount of dirty data in cache with minimal impact on application threads.&lt;/p&gt;

&lt;p&gt;To some extent, case 1 has been around forever and AFAIK none of the recent changes to eviction has made it worse.  In the past I&apos;ve tried to make forced eviction code try harder to get exclusive access to these pages but that has never been a net win.&lt;/p&gt;</comment>
                            <comment id="1378890" author="sue.loverso" created="Wed, 7 Sep 2016 20:43:09 +0000"  >&lt;p&gt;I tried to run this test for an hour.  It could only run for 43 minutes before filling the file system (which was nearly empty at the start).  It essentially ran 2x the earlier test.  It had 83 stalls of which 39 were zero drop-outs.  It completed 29 checkpoints in the 43 minutes.  There were several multi-second stalls.  I have attached &lt;tt&gt;aggressive.43min.tar.gz&lt;/tt&gt; if you want its data.&lt;/p&gt;</comment>
                            <comment id="1378414" author="sue.loverso" created="Wed, 7 Sep 2016 15:17:57 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=michael.cahill&quot; class=&quot;user-hover&quot; rel=&quot;michael.cahill&quot;&gt;michael.cahill&lt;/a&gt; I ran your branch today and things are much improved generally.  I am going to run a longer test too since the only dropouts to 0 occurred in minute 18-19.  When I ran your branch last night, there were 14 stalls, of which 10 were drops to 0.  This morning after your changes, I see 6 stalls, of which 3 are drops to 0.  Five of the six occurred in a one minute period, 18 minutes into the test, but they were not grouped together.  (I.e. it is several intermittent drops, not one longer one like yesterday&apos;s run.)  Performance is good.  This is a 20 minute standalone no-journal version of this insert test.&lt;/p&gt;

&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;th class=&apos;confluenceTh&apos;&gt; Ops/Sec/Thraad &lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt; Num Ckpts &lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt; Num Stalls &lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt; Num Zero &lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt; Long Latency &lt;/th&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt; 160 &lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt; 13 &lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt; 6 &lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt; 3 &lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt; 53.5 seconds! &lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;


&lt;p&gt;There were three super long latencies, on the order of 47-53 seconds and a handful more of about 10 seconds.  Those are troubling.&lt;/p&gt;

&lt;p&gt;Here&apos;s a chart of the entire run.  I&apos;ve uploaded this as &lt;tt&gt;run8.tar.gz&lt;/tt&gt;.  A-B is where 5 of the 6 stalls happened. &lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/137746/137746_run-branch.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;Here&apos;s the zoomed in portion A-B:&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/137745/137745_run8-zoom.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;



</comment>
                            <comment id="1375738" author="bruce.lucas@10gen.com" created="Fri, 2 Sep 2016 15:50:14 +0000"  >&lt;p&gt;Looks indeed like the stalls coincide with checkpoints, particularly with the long fsync that occurs from the beginning of the checkpoint to the point where we allocate a transaction id (marked by range of ids pinned increasing):&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/137455/137455_stalls.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;One thing that can cause this is if the journal is on the same drive as the data files - are you using separate disks for journal and data?&lt;/p&gt;</comment>
                            <comment id="1375722" author="bruce.lucas@10gen.com" created="Fri, 2 Sep 2016 15:37:48 +0000"  >&lt;blockquote&gt;
&lt;p&gt;It has been impossible to run pmp or break with gdb when stalls happen&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;I don&apos;t think it requires an unresponsive disk. I&apos;ve seen this consistently when we are fsyncing the files during a checkpoint. I believe what is happening is that the kernel does not allow the fsync call to be interrupted, which is required to attach gdb. You can check by correlating the stalls in the ftdc data with checkpoints, and more specifically the stats that record when we do the fsyncs.&lt;/p&gt;</comment>
                            <comment id="1375289" author="sue.loverso" created="Fri, 2 Sep 2016 00:23:40 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=michael.cahill&quot; class=&quot;user-hover&quot; rel=&quot;michael.cahill&quot;&gt;michael.cahill&lt;/a&gt; Here is the diagnostic data from my latest run.&lt;/p&gt;</comment>
                            <comment id="1375222" author="alexander.gorrod" created="Thu, 1 Sep 2016 22:54:00 +0000"  >&lt;p&gt;Thanks for the analysis &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sue.loverso&quot; class=&quot;user-hover&quot; rel=&quot;sue.loverso&quot;&gt;sue.loverso&lt;/a&gt;&lt;/p&gt;

&lt;blockquote&gt;&lt;p&gt;It has been impossible to run pmp or break with gdb when stalls happen&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;I&apos;ve seen two causes for this:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;There are lots and lots of threads.&lt;/li&gt;
	&lt;li&gt;The disk is unresponsive for the same time period.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Do you have statistics for the run? Could you check to see if the checkpoint sync duration approximately matches the periods where throughput stalls?&lt;/p&gt;</comment>
                            <comment id="1374898" author="sue.loverso" created="Thu, 1 Sep 2016 18:53:45 +0000"  >&lt;p&gt;Running the standalone 20 minute test with current mongo master as of Aug 30 performance is better.  Stalls still exist but checkpoints are maintained.&lt;/p&gt;

&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;th class=&apos;confluenceTh&apos;&gt; Ops/sec &lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt; Num ckpts &lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt; Stall secs &lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt; Zero stall secs&lt;/th&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt; 156 &lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt; 14 &lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt; 22 &lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt; 14 &lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;


&lt;p&gt;It has been impossible to run pmp or break with gdb when stalls happen.  It takes far longer than the stall time to connect with the process.  I was able to watch with &lt;tt&gt;perf top&lt;/tt&gt; and run &lt;tt&gt;perf record&lt;/tt&gt; every 3 seconds instead.  I was able to capture this for &lt;tt&gt;perf top&lt;/tt&gt;.  &lt;tt&gt;tree_walk_internal&lt;/tt&gt; is always at the top or 2nd.  But during a stall it spikes up to over 20%:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Normal running:&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  12.25%  libc-2.17.so        [.] __memcpy_ssse3_back&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;   5.64%  mongod              [.] __tree_walk_internal&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;   5.07%  mongod              [.] __rec_txn_read&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;   4.86%  mongod              [.] __wt_row_search&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;   4.50%  mongod              [.] __free_skip_list&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;   4.46%  mongod              [.] __wt_row_modify&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;During a stall:&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  21.38%  mongod               [.] __tree_walk_internal&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  10.14%  libc-2.17.so         [.] __memcpy_ssse3_back&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;   4.13%  mongod               [.] __rec_txn_read&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;   3.94%  mongod               [.] __wt_row_search&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;   3.93%  mongod               [.] __wt_row_modify&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Running &lt;tt&gt;perf report&lt;/tt&gt; on the perf data recorded, I see similar information during a stall.  &lt;tt&gt;perf report&lt;/tt&gt; orders things in decreasing order based on &lt;tt&gt;Children&lt;/tt&gt; percentage.  During a stall, &lt;tt&gt;tree_walk_internal&lt;/tt&gt; is the highest &lt;tt&gt;mongod&lt;/tt&gt; function with any &lt;tt&gt;self&lt;/tt&gt; time.  During normal operation, &lt;tt&gt;tree_walk_internal&lt;/tt&gt; is about the 20th &lt;tt&gt;mongod&lt;/tt&gt; function listed.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Normal:&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  Children      Self  Command        Shared Object       Symbol &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;+   12.12%    11.45%  mongod         mongod              [.] __tree_walk_internal                                                                                      &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;During stall:&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  Children      Self  Command        Shared Object       Symbol     &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;+   24.38%     0.00%  mongod         mongod              [.] __wt_thread_run                                                                                           &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;+   24.35%     0.89%  mongod         mongod              [.] __wt_evict_thread_run                                                                                     &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;+   23.44%    21.76%  mongod         mongod              [.] __tree_walk_internal                                                                                      &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="1232861" author="alexander.gorrod" created="Tue, 12 Apr 2016 06:39:40 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bruce.lucas&quot; class=&quot;user-hover&quot; rel=&quot;bruce.lucas&quot;&gt;bruce.lucas&lt;/a&gt; I have been able to reproduce similar insert throughput slowdowns during checkpoint, though it takes a lot longer for me to see the symptoms - between 10 and 15 minutes before the symptoms start. Now that I can reproduce I&apos;ll start digging deeper.&lt;/p&gt;</comment>
                            <comment id="1231812" author="milkie" created="Mon, 11 Apr 2016 13:50:54 +0000"  >&lt;p&gt;Whoops, I was indeed confused.  Thanks for the clarification.  It makes sense now.&lt;/p&gt;</comment>
                            <comment id="1231808" author="bruce.lucas@10gen.com" created="Mon, 11 Apr 2016 13:48:20 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=alexander.gorrod&quot; class=&quot;user-hover&quot; rel=&quot;alexander.gorrod&quot;&gt;alexander.gorrod&lt;/a&gt;, the run above was without slaveDelay, and was I believe due to the lagging secondary. When I did a run with slaveDelay I used 120 seconds.&lt;/p&gt;

&lt;p&gt;Yes, all the processes on the same machine (12 cores, 24 CPUs), but overall CPU utilization was well below 100%. Customer was not running with all processes on same machine.&lt;/p&gt;

&lt;p&gt;The graph above is the entire run, so less than 2 minutes to repro the issue.&lt;/p&gt;

&lt;p&gt;You might try more writer threads as the effect disappeared with a small enough number of threads. Are you seeing secondary lag in your repro? Also, cache size may matter - I was using 25 GB, customer had a much larger cache (309 GB) and saw the symptom much more dramatically.&lt;/p&gt;

&lt;p&gt;Diagnostic data attached (from a run with lag but not slaveDelay). This is a different run from the one graphed above, but parameters and outcome are the same.&lt;/p&gt;

&lt;p&gt;Regarding the possibility that there aren&apos;t many pages that are available for eviction, I&apos;m not sure how that could be the case: the workload is inserting at the end of collection, index, and oplog; and is reading from some point in the middle of the oplog. That should leave lots of pages that could be evicted, at least from a naive viewpoint about what pages could be evicted...&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=milkie&quot; class=&quot;user-hover&quot; rel=&quot;milkie&quot;&gt;milkie&lt;/a&gt;, my initial description didn&apos;t make it clear that the slowdown occurred on the primary; I&apos;ve corrected that now. Does that clarify? The delayed secondary means that it is reading from the middle of the oplog on the primary, not the end, which changes the interaction with the cache.&lt;/p&gt;</comment>
                            <comment id="1231730" author="milkie" created="Mon, 11 Apr 2016 12:29:13 +0000"  >&lt;p&gt;I don&apos;t see how slave delay could affect the behavior of caching.  The storage engine doesn&apos;t even know about any delay, and the replicated operations should be presented by the replication system to the storage engine in the same relative order, more or less.  It will be interesting to see what the underlying cause is.&lt;/p&gt;</comment>
                            <comment id="1231495" author="alexander.gorrod" created="Mon, 11 Apr 2016 04:01:28 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bruce.lucas&quot; class=&quot;user-hover&quot; rel=&quot;bruce.lucas&quot;&gt;bruce.lucas&lt;/a&gt; I attempted to reproduce this without any success.&lt;/p&gt;

&lt;p&gt;Could you give me some more information about your setup? Specifically:&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;What value did you use for slaveDelay&lt;/li&gt;
	&lt;li&gt;Where you running all of the processes on the same machine&lt;/li&gt;
	&lt;li&gt;How long did the test run before getting into the state in your graph?&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Also - would you mind uploading the diagnostic data you used to generate the graph?&lt;/p&gt;

&lt;p&gt;Re: the symptom of walking lots of pages per eviction. It is an indication that something is inefficient, but there are two different types of cause:&lt;br/&gt;
1) The eviction server algorithm is inefficient. That is it is doing a poor job of finding, choosing and queuing candidates.&lt;br/&gt;
2) The particular workload means that there aren&apos;t many pages that are available for eviction, so the server is working very hard to find the few candidates that are available.&lt;/p&gt;

&lt;p&gt;It will be interesting to see which category this falls into.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="198326">SERVER-18081</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                                        </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="198326">SERVER-18081</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="267706">SERVER-22831</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="331115">SERVER-26988</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="137406" name="23622.0901.tar.gz" size="745076" author="sue.loverso@mongodb.com" created="Fri, 2 Sep 2016 00:23:40 +0000"/>
                            <attachment id="134625" name="23622.20.tar.gz" size="1037762" author="sue.loverso@mongodb.com" created="Thu, 11 Aug 2016 15:44:55 +0000"/>
                            <attachment id="134491" name="23622.nolog.tar.gz" size="438206" author="sue.loverso@mongodb.com" created="Wed, 10 Aug 2016 19:41:36 +0000"/>
                            <attachment id="134349" name="23622.results.tar.gz" size="889127" author="sue.loverso@mongodb.com" created="Tue, 9 Aug 2016 20:03:50 +0000"/>
                            <attachment id="134709" name="2816.20.tar.gz" size="401858" author="sue.loverso@mongodb.com" created="Fri, 12 Aug 2016 02:48:35 +0000"/>
                            <attachment id="137805" name="aggressive.43min.tar.gz" size="687412" author="sue.loverso@mongodb.com" created="Wed, 7 Sep 2016 20:43:23 +0000"/>
                            <attachment id="117092" name="cs29611.png" size="130022" author="bruce.lucas@mongodb.com" created="Fri, 8 Apr 2016 20:06:08 +0000"/>
                            <attachment id="117211" name="diagnostic.data.tar" size="149504" author="bruce.lucas@mongodb.com" created="Mon, 11 Apr 2016 13:49:05 +0000"/>
                            <attachment id="134785" name="exp.tar.gz" size="424015" author="sue.loverso@mongodb.com" created="Fri, 12 Aug 2016 16:13:46 +0000"/>
                            <attachment id="117090" name="lagging.png" size="379233" author="bruce.lucas@mongodb.com" created="Fri, 8 Apr 2016 19:59:53 +0000"/>
                            <attachment id="118164" name="patch-diagnostic.data.tar" size="544256" author="bruce.lucas@mongodb.com" created="Mon, 18 Apr 2016 15:18:26 +0000"/>
                            <attachment id="118163" name="patch.png" size="228000" author="bruce.lucas@mongodb.com" created="Mon, 18 Apr 2016 15:18:26 +0000"/>
                            <attachment id="137746" name="run-branch.png" size="280744" author="sue.loverso@mongodb.com" created="Wed, 7 Sep 2016 15:11:50 +0000"/>
                            <attachment id="137745" name="run8-zoom.png" size="234453" author="sue.loverso@mongodb.com" created="Wed, 7 Sep 2016 15:11:50 +0000"/>
                            <attachment id="137747" name="run8.tar.gz" size="338845" author="sue.loverso@mongodb.com" created="Wed, 7 Sep 2016 15:18:07 +0000"/>
                            <attachment id="137455" name="stalls.png" size="106627" author="bruce.lucas@mongodb.com" created="Fri, 2 Sep 2016 15:48:20 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>15.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 11 Apr 2016 04:01:28 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        7 years, 13 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>alexander.gorrod@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            7 years, 13 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>alexander.gorrod@mongodb.com</customfieldvalue>
            <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>milkie@mongodb.com</customfieldvalue>
            <customfieldvalue>michael.cahill@mongodb.com</customfieldvalue>
            <customfieldvalue>sue.loverso@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrkazj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hsk913:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrislb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>