<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:23:28 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-50749] Re-loading is slow with py-tpcc</title>
                <link>https://jira.mongodb.org/browse/SERVER-50749</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;While loading data for py-tpcc flow control is engaged, the insert rate drops and a few inserts take 200 to 300 seconds.&lt;/p&gt;

&lt;p&gt;This is from Percona. They previously gave us the repro for &lt;a href=&quot;https://jira.mongodb.org/browse/WT-6444&quot; title=&quot;Abort a transaction if it is force evicting and oldest&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-6444&quot;&gt;&lt;del&gt;WT-6444&lt;/del&gt;&lt;/a&gt; via py-tpcc. In their report the first load into database tpcc1 takes ~20 minutes with a new mongod instance. After sleeping a few minutes and then repeating the load into database tpcc3 the second load takes ~500 minutes. They used a single-node replica set and my repro attempts do the same.&lt;/p&gt;

&lt;p&gt;Part of this is a duplicate of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-46114&quot; title=&quot;Flow-control engages on a single-node replica set&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-46114&quot;&gt;&lt;del&gt;SERVER-46114&lt;/del&gt;&lt;/a&gt; which was closed as works as designed. If you read all of the updates below, there is a chance that mongod gets stuck with flow control engaged, an insert statement that never finishes and mongod unable to shutdown. So I don&apos;t think works as designed is appropriate.&lt;/p&gt;

&lt;p&gt;Summarizing what I see below in my repro attempts:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;this problem is new in 4.4.0. I tried but could not reproduce this with 4.2.9.&lt;/li&gt;
	&lt;li&gt;many inserts take more than 5 seconds with 4.4.0 (up to 390 seconds ignoring the hang). No inserts take more than 5 seconds with 4.2.9&lt;/li&gt;
	&lt;li&gt;in one test mongod got stuck. An insert statement was saturating a CPU core but making no progress for 1+ hour. It did not stop after killOp(). Shutting down mongod via &quot;killall mongod&quot; did not stop mongod and eventually I did kill -9.&lt;/li&gt;
	&lt;li&gt;with flow control enabled and 4.4.0 there are stalls (inserts that take 10 to 390 seconds)&lt;/li&gt;
	&lt;li&gt;with flow control disabled and 4.4.0 there are still stalls, but they are not as bad (10 to 60 seconds) as above&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;I have ftdc and mongod error logs for most of the results listed below. I can provide them if requested. There are many, so I prefer to do that on demand.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1462046">SERVER-50749</key>
            <summary>Re-loading is slow with py-tpcc</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="backlog-server-storage-engines">Backlog - Storage Engines Team</assignee>
                                    <reporter username="mark.callaghan@mongodb.com">Mark Callaghan</reporter>
                        <labels>
                            <label>KP44</label>
                            <label>dmd-escapes</label>
                            <label>perf-escapes</label>
                    </labels>
                <created>Thu, 3 Sep 2020 15:29:58 +0000</created>
                <updated>Tue, 6 Dec 2022 02:09:55 +0000</updated>
                            <resolved>Fri, 12 Nov 2021 05:47:44 +0000</resolved>
                                    <version>4.4.0</version>
                                                                        <votes>0</votes>
                                    <watches>17</watches>
                                                                                                                <comments>
                            <comment id="4184865" author="sulabh.mahajan" created="Fri, 12 Nov 2021 05:47:19 +0000"  >&lt;p&gt;I confirmed the problem goes away if we do not use the unique index. Actually current master performs better with tpcc load and reload than 4.2.17. Please see &lt;a href=&quot;https://jira.mongodb.org/browse/WT-6664&quot; title=&quot;Cache eviction causes high latency during py-tpcc.&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-6664&quot;&gt;&lt;del&gt;WT-6664&lt;/del&gt;&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/browse/WT-6664?focusedCommentId=4184861&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-4184861&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;this comment&lt;/a&gt;, for more details.&lt;/p&gt;

&lt;p&gt;Given there is nothing new for this server ticket, I am going to mark it dup of &lt;a href=&quot;https://jira.mongodb.org/browse/WT-6664&quot; title=&quot;Cache eviction causes high latency during py-tpcc.&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-6664&quot;&gt;&lt;del&gt;WT-6664&lt;/del&gt;&lt;/a&gt;. &lt;a href=&quot;https://jira.mongodb.org/browse/WT-6664&quot; title=&quot;Cache eviction causes high latency during py-tpcc.&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-6664&quot;&gt;&lt;del&gt;WT-6664&lt;/del&gt;&lt;/a&gt; will likely be duped to another WiredTiger ticket where we are resolving these issues with unique indexes.&lt;/p&gt;</comment>
                            <comment id="4120267" author="sulabh.mahajan" created="Tue, 12 Oct 2021 22:47:24 +0000"  >&lt;p&gt;Adding &lt;a href=&quot;https://jira.mongodb.org/browse/WT-6664?focusedCommentId=4117212&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-4117212&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;the comment describing the current state of the issue&lt;/a&gt; from &lt;a href=&quot;https://jira.mongodb.org/browse/WT-6664&quot; title=&quot;Cache eviction causes high latency during py-tpcc.&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-6664&quot;&gt;&lt;del&gt;WT-6664&lt;/del&gt;&lt;/a&gt;:&lt;/p&gt;

&lt;p&gt;Here is the summary of the current state of the things:&lt;/p&gt;

&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;Version&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;FlowControl&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;Load 1&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;Load 2&lt;/th&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;4.2.17&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;ON&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;79m 33s&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;81m 41s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;4.2.17&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;OFF&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;86m 21s&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;86m 35s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;Latest Master (79370f1ea8)&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;ON&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;Killed after 120m&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;N/A&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;Latest Master (79370f1ea8)&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;OFF&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;97m 59s&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;Crash after 1539m 54s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;
</comment>
                            <comment id="3398979" author="mark.callaghan" created="Thu, 17 Sep 2020 13:36:40 +0000"  >&lt;p&gt;Repeating a comment from wt-6664. Percona, the reporter of this problem, now has a &lt;a href=&quot;https://www.percona.com/blog/2020/09/16/mongodb-4-4-performance-regression-overwhelmed-by-memory&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;blog post&lt;/a&gt;. As reported above, the problem for them doesn&apos;t happen or is much less likely with a smaller WiredTiger cache.&lt;/p&gt;</comment>
                            <comment id="3388728" author="keith.smith" created="Fri, 11 Sep 2020 20:18:33 +0000"  >&lt;p&gt;Based on &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mark.callaghan&quot; class=&quot;user-hover&quot; rel=&quot;mark.callaghan&quot;&gt;mark.callaghan&lt;/a&gt;&apos;s comment that he&apos;s using a system with 60GB of RAM, I did a few more tests on a similar system using different size WT caches. &#160;It looks like this problem gets &lt;b&gt;worse&lt;/b&gt; with larger cache sizes. &#160;See &lt;a href=&quot;https://jira.mongodb.org/browse/WT-6664&quot; title=&quot;Cache eviction causes high latency during py-tpcc.&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-6664&quot;&gt;&lt;del&gt;WT-6664&lt;/del&gt;&lt;/a&gt; for details.&lt;/p&gt;</comment>
                            <comment id="3385787" author="geert.bosch" created="Thu, 10 Sep 2020 17:20:30 +0000"  >&lt;p&gt;Note that all of this worked before. The case with the cache stuck was &lt;em&gt;not&lt;/em&gt; stuck due to flow control, but due to an issue with the history store. All time was spent in the history store, see the backtraces. That thread already was past the flow-control state (it had acquired the global lock) and should have run until the next interruption point, but never did. That&apos;s a WT issue. Now, I can see how in this case flow-control compounded a problem, and we can fix that, but it doesn&apos;t fix the underlying WT issue. Similarly, inserting 1 doc at a time wastes enough CPU in mongod that the load on the storage engine will drop, also avoiding the cache stuck issue. That doesn&apos;t seem a good solution in the long run. Why does mongod behave so much worse with 4.4 than with 4.2?&lt;/p&gt;</comment>
                            <comment id="3385777" author="keith.smith" created="Thu, 10 Sep 2020 17:16:52 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bruce.lucas&quot; class=&quot;user-hover&quot; rel=&quot;bruce.lucas&quot;&gt;bruce.lucas&lt;/a&gt;: Your assumption is correct. &quot;flowControl isLagged&quot; was zero throughout my run with a small batch size. Likewise, the WT cache was much happier. It never entered aggressive eviction mode, never used application threads for eviction, and rarely had to do in-memory restoration on evicted pages (1-2/sec). &#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="3385680" author="bruce.lucas@10gen.com" created="Thu, 10 Sep 2020 16:40:09 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=keith.smith&quot; class=&quot;user-hover&quot; rel=&quot;keith.smith&quot;&gt;keith.smith&lt;/a&gt;, the batch size is expected to affect both cache pinning and operation of flow control if it engages. I imagine that the beneficial effect in your test has been because of the former, and there will no longer be lag so flow control will no longer be engaging, but it may be worth checking that assumption in the ftdc by checking that the flow control isLagged metric remains 0.&lt;/p&gt;</comment>
                            <comment id="3385656" author="mark.callaghan" created="Thu, 10 Sep 2020 16:26:43 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bruce.lucas&quot; class=&quot;user-hover&quot; rel=&quot;bruce.lucas&quot;&gt;bruce.lucas&lt;/a&gt;&#160;- I use a c3.8xlarge host setup by DSI (16 cores, 60G of RAM, 20k EBS IOPs), benchmark client shared the host and benchmark client uses 10 connections (and 8 threads or processes)&lt;/p&gt;</comment>
                            <comment id="3385651" author="mark.callaghan" created="Thu, 10 Sep 2020 16:24:13 +0000"  >&lt;p&gt;Added:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;flow0.mo440.tar.gz, flow0.mo429.tar.gz - flow control disabled for 4.4.0 and 4.2.9&lt;/li&gt;
	&lt;li&gt;flow1.mo440.tar.gz, flow1.mo429.tar.gz - flow control enabled for 4.4.0 and 4.2.9&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;These include ftdc, mongod error log and test output. The directory structure is like:&lt;br/&gt;
flow0.mo440/v1, flow0.mo440.v2 -&amp;gt; v1 and v2 because I repeated tests&lt;/p&gt;

&lt;p&gt;from &quot;ls flow0.mo440/v1&quot;&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;w100 -&amp;gt; w400 is the number of warehouses (from 100 up to 400 in some cases)&lt;/li&gt;
	&lt;li&gt;pypyyes -&amp;gt; the python script was run by pypy, not python&lt;/li&gt;
	&lt;li&gt;l10 -&amp;gt; 10 loader clients (connections)&lt;/li&gt;
&lt;/ul&gt;


&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;r.w100.l10.pypyyes  r.w150.l10.pypyyes  r.w200.l10.pypyyes  r.w250.l10.pypyyes  r.w300.l10.pypyyes  r.w350.l10.pypyyes  r.w400.l10.pypyyes&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;



&lt;p&gt;from &quot;ls flow1.mo440/v2/r.w100.l10.pypyyes&quot;&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;h.* and f.* files are PMP output&lt;/li&gt;
	&lt;li&gt;you can ignore o.* files&lt;/li&gt;
	&lt;li&gt;mongod.log is mongod error log from test end&lt;/li&gt;
&lt;/ul&gt;


&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;diagnostic.data  f.200903_210423  h.200903_204618  h.200903_211703  o.asz1  o.grep  o.lsh   o.sz1  op.1   op.12  op.15  op.18  op.20  op.5  op.8&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;f.200903_204618  f.200903_211703  h.200903_204811  h.200903_211944  o.asz2  o.io    o.lshr  o.sz2  op.10  op.13  op.16  op.19  op.3   op.6  op.9&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;f.200903_204811  f.200903_211944  h.200903_210423  mongod.log       o.dbs   o.ls    o.ps    o.vm   op.11  op.14  op.17  op.2   op.4   op.7&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="3385646" author="keith.smith" created="Thu, 10 Sep 2020 16:23:11 +0000"  >&lt;p&gt;I reran my test with internalInsertMaxBatchSize=2.&lt;/p&gt;

&lt;p&gt;The maximum latency during the test dropped from 33.5 to 1.4 seconds. So I&apos;ll go out on a limb and assert that this parameter affects the performance &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;

&lt;p&gt;I can experiment with other values, but I worry that the results would be dependent on workload and system. We&apos;d find a value that works on a few tests here, but some customer will discover a workload where it blows up. Is there a principled way we can reason about the tradeoffs here? &#160;Does this have to be a constant, or would it make sense to dynamically adjust the batch size based on the state of the system?&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mark.callaghan&quot; class=&quot;user-hover&quot; rel=&quot;mark.callaghan&quot;&gt;mark.callaghan&lt;/a&gt;: I&apos;m already using &lt;tt&gt;pypy.&lt;/tt&gt;&lt;/p&gt;</comment>
                            <comment id="3385609" author="bruce.lucas@10gen.com" created="Thu, 10 Sep 2020 16:10:57 +0000"  >&lt;p&gt;Thanks Mark. That does look quite a bit different from the data Keith posted (although it may well be simply a more extreme manifestation of the same underlying problems).&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;lag continues growing throughout the test and does not recover after the test, consistent with a hang&lt;/li&gt;
	&lt;li&gt;flow control is constantly engaged&lt;/li&gt;
	&lt;li&gt;flow control is actually doing significant throttling in this case because the rate of global lock acquisition before flow control was ~500/s, so throttling to the floor of 100/s is signficant&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;But given the hang that appears to create unrecoverable cache pressure, I&apos;m not sure that the details of the behavior of flow control or the latencies aside from the hang in this particular test are too interesting as the hang is by far the worse problem and can be expected to cause the other problems.&lt;/p&gt;

&lt;p&gt;In addition to the python difference you noted, your data shows a 4x larger server running mongod than Keith&apos;s data.&lt;/p&gt;</comment>
                            <comment id="3385552" author="mark.callaghan" created="Thu, 10 Sep 2020 15:56:42 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bruce.lucas&quot; class=&quot;user-hover&quot; rel=&quot;bruce.lucas&quot;&gt;bruce.lucas&lt;/a&gt; Added the missing ftdc and error log files I mentioned &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-50749?focusedCommentId=3375748&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-3375748&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=keith.smith&quot; class=&quot;user-hover&quot; rel=&quot;keith.smith&quot;&gt;keith.smith&lt;/a&gt; - please use pypy rather than python to run py-tpcc as that has much lower CPU overhead leading to more pressure on MongoDB&lt;/p&gt;

&lt;p&gt;Next up is ftdc with and without flow control enabled&lt;/p&gt;</comment>
                            <comment id="3385404" author="bruce.lucas@10gen.com" created="Thu, 10 Sep 2020 15:10:55 +0000"  >&lt;p&gt;This example seems representative:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/277709/277709_example.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;D: flow control engages, begins doing some throttling (&quot;flowControl timeAcquiringMicros&quot;), but doesn&apos;t really have much effect on insert rates (&quot;opcounters insert&quot;, &quot;cursor insert calls&quot;). I think this is because of another known issue with flow control: throttling is based on global lock acquisition, but because of batching global locks are being acquired at a low rate, average 135/s (&quot;locks Global acquireCount w&quot;); but in order to avoid potential impact to internal operations flow control has a throttling floor of 100 lock acquisitions /s, so it&apos;s not going to do much throttling if the average without throttling is only 135/s.&lt;/li&gt;
&lt;/ul&gt;


&lt;ul&gt;
	&lt;li&gt;E: cache pressure takes over as the throttling mechanism (&quot;application thread time waiting for cache&quot;). Note though that even though we&apos;ve started throttling quite heavily (based on insert rates), lag is still increasing. I think this might be the effect I mentioned above. However here it doesn&apos;t really matter because at this point it is cache pressure that&apos;s doing the throttling, not flow control, so it doesn&apos;t really matter that flow control remains engaged.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;In addition to helping in the way that Eric describes, decreasing the batch size will increase the rate of lock acquisition so should make flow control more effective if it does engage; tbd whether this is a good thing or not.&lt;/p&gt;

&lt;p&gt;Another question is if flow control is not doing a lot, why is it having such a large impact on maximum latency? I seem to recall some concern about whether the o/s mechanism underlying flow control tickets is fair, or may even be intentionally unfair to favor throughput over latency. If unfair, maybe a thread could repeatedly get starved out trying to obtain a ticket?&lt;/p&gt;</comment>
                            <comment id="3385152" author="keith.smith" created="Thu, 10 Sep 2020 13:42:50 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=milkie&quot; class=&quot;user-hover&quot; rel=&quot;milkie&quot;&gt;milkie&lt;/a&gt;: It should be easy enough to re-run. &#160;I&apos;ll try first with internalInsertMaxBatchSize set very low (1?), so we can test the hypothesis that this is causing the stalls. &#160;If it is, then we can experiment with a better size. &#160;&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=geert.bosch&quot; class=&quot;user-hover&quot; rel=&quot;geert.bosch&quot;&gt;geert.bosch&lt;/a&gt;: Flow control and history store also seem implicated in my test. I was just focused on the WT cache when I selected the stats to look at. &#160;See below:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/277684/277684_Screen+Shot+2020-09-10+at+9.35.27+AM.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &#160;&lt;/p&gt;</comment>
                            <comment id="3384993" author="milkie" created="Thu, 10 Sep 2020 12:29:37 +0000"  >&lt;p&gt;Alex, for your number 1 above, the current batch size for insert is &quot;500&quot; and this is run-time settable with the &quot;internalInsertMaxBatchSize&quot; server parameter, so we can experiment with different values to see their effect.  &lt;br/&gt;
Can someone rerun the test with internalInsertMaxBatchSize set to something like 100?  Running with five times the number of individual transactions as before shouldn&apos;t have a material effect on overall runtime (I surmise), but with open transactions five times smaller should enable WiredTiger to more efficiently evict pages as new data is inserted.  Once we arrive at a better max batch size, we can consider changing the default in released MongoDB.&lt;/p&gt;</comment>
                            <comment id="3384688" author="alexander.gorrod" created="Thu, 10 Sep 2020 06:55:40 +0000"  >&lt;p&gt;I believe &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=keith.smith&quot; class=&quot;user-hover&quot; rel=&quot;keith.smith&quot;&gt;keith.smith&lt;/a&gt; reproduced this. We walked through the FTDC in detail today.&lt;/p&gt;

&lt;p&gt;The issue that introduces latencies with this test is that it batches operations into large groups, and executes multiple of those batches in parallel. That results in MongoDB creating multiple large transactions running against WiredTiger at the same time. Sometimes one of those transactions falls behind which leads to the oldest timestamp and transaction ID being pinned. Since there is lots of content pinned by active transactions (the content WiredTiger still can&apos;t evict even with durable history), that leads to WiredTiger choosing inefficient pages to evict when trying to find enough space in cache to complete the lagging transaction. Our team did work to alleviate the long latencies seen in this workload coming into the 4.4.0 release, but that tuning either didn&apos;t catch enough cases or was undone by another change (which is possible - the heuristics we added to handle the case were tricky to get right).&lt;/p&gt;

&lt;p&gt;The reason we split off a separate WiredTiger ticket is that the server could change its behavior to alleviate the symptom here as well. I can see two avenues that would be worth exploring:&lt;/p&gt;

&lt;p&gt;1) It could implement a smarter version of &lt;tt&gt;insertBatchAndHandleErrors&lt;/tt&gt;. At the moment that function tries to do all operations in a batch within a single WiredTiger transaction. If that fails it falls back to doing each operation in a separate transaction - WiredTiger tries very hard not to fail a transaction due to the amount of resources it holds. Reducing the batch size in that code would help ensure transactions didn&apos;t get big enough to cause the sort of pressure being put on the storage engine by the workload. &lt;br/&gt;
2) Fixing the issue that causes flow control to exacerbate the poor behavior. Since running with flow control endabled extends the maximum latency for an operation from 80 seconds to 390 seconds (if I&apos;m reading the results correctly).&lt;/p&gt;</comment>
                            <comment id="3384585" author="geert.bosch" created="Thu, 10 Sep 2020 03:06:13 +0000"  >&lt;p&gt;Both flow-control activating (supposedly due to a lagging majority-committed point), and in the hanging case the stalling threads having backtraces in the history store seem to point to a scenario that may lead to specific operations needing access to the history store not making progress, resulting in a negative feedback loop. It doesn&apos;t look like your experiment reproduced that scenario.&lt;/p&gt;</comment>
                            <comment id="3384283" author="keith.smith" created="Wed, 9 Sep 2020 22:01:20 +0000"  >&lt;p&gt;I&apos;ve been reproducing this with partial success using &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mark.callaghan&quot; class=&quot;user-hover&quot; rel=&quot;mark.callaghan&quot;&gt;mark.callaghan&lt;/a&gt;&apos;s repro recipe. &#160;He performs two runs of py-tpcc and observes the second take much longer. &#160;I do not see this. &#160;But like him I see a number of high latency insert operations. &#160;It looks like WT cache eviction is a (the?) culprit here.&lt;/p&gt;

&lt;p&gt;Looking at the FTDC data from my test, I added a pin at the time of every request that took longer than 16s. &#160;That was 15 requests in about 25 minutes of run time. &#160;These request align perfectly with spikes in various (mostly cache related) WT stats:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/277639/277639_Screen+Shot+2020-09-09+at+6.16.19+PM.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;FWIW, I&apos;ve been doing this on a VM with 16 GB of RAM and four cores.&lt;/p&gt;

&lt;p&gt;I&apos;ve opened &lt;a href=&quot;https://jira.mongodb.org/browse/WT-6664&quot; title=&quot;Cache eviction causes high latency during py-tpcc.&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-6664&quot;&gt;&lt;del&gt;WT-6664&lt;/del&gt;&lt;/a&gt; for this. But since I haven&apos;t reproduced the original issue of the &lt;tt&gt;py-tpcc&lt;/tt&gt; reload being so much slower I&apos;m hesitant to say this is the entire cause for this problem.&lt;/p&gt;</comment>
                            <comment id="3380310" author="bruce.lucas@10gen.com" created="Tue, 8 Sep 2020 17:07:21 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mark.callaghan&quot; class=&quot;user-hover&quot; rel=&quot;mark.callaghan&quot;&gt;mark.callaghan&lt;/a&gt;, I&apos;m not seeing these two files on the ticket:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;ftdc.tpcc.440.hang.tar - diagnostic.data directory&lt;br/&gt;
mongod.log.440.hang.gz - the mongod log file&lt;/p&gt;&lt;/blockquote&gt;</comment>
                            <comment id="3376242" author="mark.callaghan" created="Fri, 4 Sep 2020 16:21:47 +0000"  >&lt;p&gt;Results similar to the previous comment, but with flow control disabled. Stalls were not as bad, but there were still many inserts that take more than 5 seconds with 4.4.0 and the worst case was ~75 seconds. No inserts take &amp;gt;= 5 seconds for 4.2.9.&lt;/p&gt;

&lt;p&gt;Legend:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;L1, L2 - number of seconds for the load&lt;/li&gt;
	&lt;li&gt;slow - number of slow queries with slowms=5000&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;flow control disabled&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        4.4.0 --------- 4.4.0   4.2.9 --------- 4.2.9&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;w       L1      L2      slow    L1      L2      slow&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;100     203     193     12      186     189     0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;150     306     339     33      278     274     0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;200     413     377     37      365     373     0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;250     486     555     49      461     458     0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;300     615     626     60      560     560     0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;350     734     673     47      652     663     0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;401     799     808     74      749     727     0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Response time histogram for slow queries from the test with 4.4.0 and flow&lt;br/&gt;
control disabled. These are counted from the slow query log with slowms=5000.&lt;br/&gt;
Even with flow control disabled, stalls are still a problem for WiredTiger.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;w       &amp;lt;=10s   &amp;lt;=20s   &amp;lt;=30s   &amp;lt;=40s   &amp;lt;=50s   &amp;lt;=60s   &amp;lt;=70s   &amp;lt;=80s&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;100     8       2       1       1&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;150     19      8       3       1               2&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;200     30      2       1       3       1&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;250     32      8       5       3                               1&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;300     42      8       3       4       1       2&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;350     30      6       5       6&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;400     54      11      5       1       3&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="3376240" author="mark.callaghan" created="Fri, 4 Sep 2020 16:18:57 +0000"  >&lt;p&gt;I repeated tests for 100, 150, 200, 250, 300, 350 and 400 warehouses. For all of these the database was in memory &#8211; no reads from storage. Tests were done for 4.4.0 and 4.2.9 with and without flow control enabled.&lt;/p&gt;

&lt;p&gt;First up is results with flow control enabled.&lt;/p&gt;

&lt;p&gt;Legend:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;L1, L2 - number of seconds for the load&lt;/li&gt;
	&lt;li&gt;slow - number of slow queries with slowms=5000&lt;/li&gt;
	&lt;li&gt;inf == mongod was stuck, killed after 1 hour&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;flow control enabled, v1. The &lt;b&gt;X&lt;/b&gt; indicates that I was unable to count slow queries.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        4.4.0 --------- 4.4.0   4.2.9 --------- 4.2.9&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;w       L1      L2      slow    L1      L2      slow&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;100     184     444     X       180     187     0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;150     297     621     X       275     269     0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;200     540     591     X       368     365     0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;250     775     653     X       459     466     0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;300     711     711     X&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;flow control enabled, v2. The &lt;b&gt;inf&lt;/b&gt; indicates that an insert statement and mongod got stuck as described above. I ended up having to do kill -9 to stop mongod.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        4.4.0 --------- 4.4.0   4.2.9 --------- 4.2.9&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;w       L1      L2      slow    L1      L2      slow&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;100     175     inf     X&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;flow control enabled, v3. There were no inserts that took &amp;gt;= 5 seconds with 4.2.9, but many with 4.4.0&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        4.4.0 --------- 4.4.0   4.2.9 --------- 4.2.9&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;w       L1      L2      slow    L1      L2      slow&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;100     180     248      9      183     190     0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;150     273     306      6      269     273     0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;200     444     387      9      362     371     0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;250     720     601     24      449     466     0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;300     761     992     34      547     573     0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;350     710     671      7      664     658     0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;400     924     915     36      754     761     0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Response time histogram for slow queries from the test with 4.4.0 and flow&lt;br/&gt;
control enabled. These are counted from the slow query log with slowms=5000.&lt;br/&gt;
The problem of stalls from flow control is intermittent and in this run&lt;br/&gt;
w=250 and w=300 were the worst.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;w       &amp;lt;=10s   &amp;lt;=20s   &amp;lt;=30s   &amp;lt;=40s   &amp;lt;=50s   &amp;lt;=60s   &amp;lt;=70s   &amp;lt;=80s&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;100      8                                                      1&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;150      1       4                      1&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;200      4       1      2               1               1&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;250     16       2      3       1                                       1 @ 150s, 1 @ 270s&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;300     19       9      3       1       1               1               1 @ 160s, 1 @ 180s, 1 @ 390s&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;350      2       2      3                                               1 @ 110s&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;400     12      12      4       3       1               7               2 @ 110s&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="3375764" author="mark.callaghan" created="Thu, 3 Sep 2020 22:06:36 +0000"  >&lt;p&gt;Have yet to reproduce this in 4.2.9. It is intermittent with 4.4.0.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="1915961">SERVER-61185</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="1468424">WT-6664</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="1468424">WT-6664</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="277639" name="Screen Shot 2020-09-09 at 6.16.19 PM.png" size="161070" author="keith.smith@mongodb.com" created="Wed, 9 Sep 2020 22:19:23 +0000"/>
                            <attachment id="277684" name="Screen Shot 2020-09-10 at 9.35.27 AM.png" size="85015" author="keith.smith@mongodb.com" created="Thu, 10 Sep 2020 13:48:38 +0000"/>
                            <attachment id="277709" name="example.png" size="328092" author="bruce.lucas@mongodb.com" created="Thu, 10 Sep 2020 15:09:38 +0000"/>
                            <attachment id="277743" name="flow0.mo429.tar.gz" size="4531399" author="mark.callaghan@mongodb.com" created="Thu, 10 Sep 2020 16:12:15 +0000"/>
                            <attachment id="277742" name="flow0.mo440.tar.gz" size="8395229" author="mark.callaghan@mongodb.com" created="Thu, 10 Sep 2020 16:11:42 +0000"/>
                            <attachment id="277741" name="flow1.mo429.tar.gz" size="6458256" author="mark.callaghan@mongodb.com" created="Thu, 10 Sep 2020 16:10:17 +0000"/>
                            <attachment id="277738" name="flow1.mo440.tar.gz" size="8517528" author="mark.callaghan@mongodb.com" created="Thu, 10 Sep 2020 16:09:14 +0000"/>
                            <attachment id="277725" name="ftdc.tpcc.440.hang.tar" size="1382400" author="mark.callaghan@mongodb.com" created="Thu, 10 Sep 2020 15:50:55 +0000"/>
                            <attachment id="277681" name="metrics.2020-09-09T20-02-23Z-00000" size="1636895" author="keith.smith@mongodb.com" created="Thu, 10 Sep 2020 13:44:35 +0000"/>
                            <attachment id="277724" name="mongod.log.440.hang.gz" size="19164" author="mark.callaghan@mongodb.com" created="Thu, 10 Sep 2020 15:50:49 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>22.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25135"><![CDATA[Storage Engines]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="22495"><![CDATA[v5.1]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 3 Sep 2020 19:40:54 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 12 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>alexander.golin@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 12 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>alexander.gorrod@mongodb.com</customfieldvalue>
            <customfieldvalue>backlog-server-storage-engines</customfieldvalue>
            <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>milkie@mongodb.com</customfieldvalue>
            <customfieldvalue>geert.bosch@mongodb.com</customfieldvalue>
            <customfieldvalue>keith.smith@mongodb.com</customfieldvalue>
            <customfieldvalue>mark.callaghan@mongodb.com</customfieldvalue>
            <customfieldvalue>sulabh.mahajan@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hy3rwv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr3h33:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="5377">Storage - Ra 2021-09-20</customfieldvalue>
    <customfieldvalue id="5408">Storage - Ra 2021-10-04</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hy3e67:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>