<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:40:31 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-16269] WiredTiger blocks queries and updates for many seconds.</title>
                <link>https://jira.mongodb.org/browse/SERVER-16269</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Comparing MMAP to WiredTiger on an 8 core server with 32GB Ram and local SSD (AWS Instance).&lt;/p&gt;

&lt;p&gt;Throwing a mix of insert/update/query at the server and wiredtiger viewed in mongostat shows really inconsistent performance - also many operations in log are poking about the 100ms mark with inserts taking &amp;gt;1s in some cases. I dont see this inconsistency with mmap v1.&lt;/p&gt;

&lt;p&gt;mmapv1 engine  output from mongostat looks like this.&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;insert query update delete getmore command flushes mapped vsize    res faults     locked db idx miss % qr|qw ar|aw netIn netOut conn     time&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  3076  8761   5582     *0       0     2|0       0   6.0g 12.5g 415.0m      0         :0.0%          0 37|85 42|86   10m     9m  129 15:33:16&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  2789 10304   5979     *0       0     1|0       0   6.0g 12.5g 418.0m      0         :0.0%          0 42|82 44|83   11m    11m  129 15:33:17&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  3668  8509   4778     *0       0     1|0       0   6.0g 12.5g 412.0m      0         :0.0%          0 10|65 39|67   10m     9m  129 15:33:18&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  4151  8757   5172     *0       0     1|0       0   6.0g 12.5g 472.0m      0         :0.0%          0 29|94 32|95   11m     9m  129 15:33:19&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  3019  7988   6096     *0       0     1|0       0   6.0g 12.5g 477.0m      0         :0.0%          0 44|78 48|79   11m     9m  129 15:33:20&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  3661  7152   5647     *0       0     2|0       0   6.0g 12.5g 478.0m      0         :0.0%          0 53|75 56|75   11m     8m  129 15:33:21&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  2378  7132   6574     *0       0     1|0       0   6.0g 12.5g 479.0m      0         :0.0%          0 32|91 35|92   11m     8m  129 15:33:22&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  2437  8355   6379     *0       0     1|0       0   6.0g 12.5g 484.0m      0         :0.0%          0 41|83 43|84   11m     9m  129 15:33:23&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  2334  8889   5756     *0       0     1|0       0   6.0g 12.5g 483.0m      0         :0.0%          0 52|72 54|73   10m     9m  129 15:33:24&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;WT looks like this&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;insert query update delete getmore command flushes mapped vsize res faults     locked db idx miss % qr|qw ar|aw netIn netOut conn     time&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  2348  8786   5922     *0       0     2|0     n/a    n/a   n/a n/a    n/a           n/a        n/a   n/a   n/a   10m     9m  129 15:35:48&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  9918 30670  21542     *0       0     1|0     n/a    n/a   n/a n/a    n/a           n/a        n/a   n/a   n/a   39m    34m  129 15:35:49&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; 11902 33152  20733     *0       0     1|0     n/a    n/a   n/a n/a    n/a           n/a        n/a   n/a   n/a   41m    37m  129 15:35:50&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  3801  8540   5223     *0       0     1|0     n/a    n/a   n/a n/a    n/a           n/a        n/a   n/a   n/a   11m     9m  129 15:35:51&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    *0    *0     *0     *0       0     1|0     n/a    n/a   n/a n/a    n/a           n/a        n/a   n/a   n/a   79b    13k  129 15:35:52&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  2934  7013   5251     *0       0     2|0     n/a    n/a   n/a n/a    n/a           n/a        n/a   n/a   n/a   10m     8m  129 15:35:53&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;   548  1183    896     *0       0     1|0     n/a    n/a   n/a n/a    n/a           n/a        n/a   n/a   n/a    1m     1m  129 15:35:54&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    52    91     77     *0       0     1|0     n/a    n/a   n/a n/a    n/a           n/a        n/a   n/a   n/a  159k   118k  129 15:35:55&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  5622 14617  12105     *0       0     1|0     n/a    n/a   n/a n/a    n/a           n/a        n/a   n/a   n/a   22m    16m  129 15:35:56&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</description>
                <environment></environment>
        <key id="170663">SERVER-16269</key>
            <summary>WiredTiger blocks queries and updates for many seconds.</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="michael.cahill@mongodb.com">Michael Cahill</assignee>
                                    <reporter username="john.page@mongodb.com">John Page</reporter>
                        <labels>
                            <label>wiredtiger</label>
                    </labels>
                <created>Fri, 21 Nov 2014 15:37:38 +0000</created>
                <updated>Fri, 3 Dec 2021 19:34:36 +0000</updated>
                            <resolved>Mon, 29 Dec 2014 05:52:31 +0000</resolved>
                                    <version>2.8.0-rc0</version>
                                                    <component>Storage</component>
                                        <votes>2</votes>
                                    <watches>24</watches>
                                                                                                                <comments>
                            <comment id="794448" author="michael.cahill" created="Mon, 29 Dec 2014 05:52:31 +0000"  >&lt;p&gt;This has been split into &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-16651&quot; title=&quot;Mass deletions and updates (including TTL) completely starve insertions in replica set under WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-16651&quot;&gt;&lt;del&gt;SERVER-16651&lt;/del&gt;&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-16662&quot; title=&quot;Extended pauses in WiredTiger when workers greatly exceeds number of cores&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-16662&quot;&gt;&lt;del&gt;SERVER-16662&lt;/del&gt;&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-16667&quot; title=&quot;Isolated writes with extreme latencies in replica set under WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-16667&quot;&gt;&lt;del&gt;SERVER-16667&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="793383" author="dan@10gen.com" created="Tue, 23 Dec 2014 22:17:11 +0000"  >&lt;p&gt;We&apos;ve seen that disabling Transparent Huge Pages makes the &quot;pauses&quot; go away in some workloads.  In order to do, the user had to also change &lt;tt&gt;/sys/kernel/mm/transparent_hugepage/defrag&lt;/tt&gt; to &lt;tt&gt;never&lt;/tt&gt;.&lt;br/&gt;
See also: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-16643&quot; title=&quot;extend existing THP warning to THP defrag setting&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-16643&quot;&gt;&lt;del&gt;SERVER-16643&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Keeping ticket open for further analysis.&lt;/p&gt;</comment>
                            <comment id="793030" author="keith.bostic" created="Tue, 23 Dec 2014 16:08:54 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=asya&quot; class=&quot;user-hover&quot; rel=&quot;asya&quot;&gt;asya&lt;/a&gt;, at this point, are you running with the sched_yield calls backing off to sleep, or with the the sched_yield calls just spinning?&lt;/p&gt;</comment>
                            <comment id="792678" author="keith.bostic" created="Mon, 22 Dec 2014 23:09:41 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bruce.lucas%4010gen.com&quot; class=&quot;user-hover&quot; rel=&quot;bruce.lucas@10gen.com&quot;&gt;bruce.lucas@10gen.com&lt;/a&gt;: all of our tests were on AWS Linux machines.&lt;/p&gt;</comment>
                            <comment id="791679" author="asya" created="Sun, 21 Dec 2014 16:52:10 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=robert.j.moore%40allanbank.com&quot; class=&quot;user-hover&quot; rel=&quot;robert.j.moore@allanbank.com&quot;&gt;robert.j.moore@allanbank.com&lt;/a&gt; thank you for the offer - I also can reproduce this with YCSB (and suspect a related ticket I filed is also this as a root cause).   Fewer threads in fact show less of this problem but at the cost of lower overall throughput.  I&apos;m testing a couple of internal configurations and  settings to determine which path may be most likely to alleviate this.&lt;/p&gt;</comment>
                            <comment id="791660" author="robert.j.moore@allanbank.com" created="Sun, 21 Dec 2014 14:18:14 +0000"  >&lt;p&gt;Asya -&lt;/p&gt;

&lt;p&gt;I am seeing these same pauses using YCSB (&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-16619&quot; title=&quot;Very long pauses using 2.8.0-rc3 when running YCSB workload B.&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-16619&quot;&gt;&lt;del&gt;SERVER-16619&lt;/del&gt;&lt;/a&gt;) and I have seen them with 1 connections up to 50 connections.&lt;/p&gt;

&lt;p&gt;Where is a log with 1 connection:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2014-12-20T18:33:29.414-0500 I WRITE    [conn6093] update ycsb.usertable query: { _id: &quot;user7906682381250086252&quot; } update: { $set: { field9: BinData(0, 2229303E3C30332C3D392B3D273A212B303C3738213D3A35383D323935292025263B252A3E35363C3D203F3D342E29202331253E3926343537223A3033312A35312334383529...) } } nscanned:1 nscannedObjects:1 nMatched:1 nModified:1 fastmod:1 keyUpdates:0  1310ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2014-12-20T18:33:29.414-0500 I QUERY    [conn6093] command ycsb.$cmd command: update { update: &quot;usertable&quot;, ordered: false, writeConcern: { w: 1 }, updates: [ { q: { _id: &quot;user7906682381250086252&quot; }, u: { $set: { field9: BinData(0, 2229303E3C30332C3D392B3D273A212B303C3738213D3A35383D323935292025263B252A3E35363C3D203F3D342E29202331253E3926343537223A3033312A35312334383529...) } } } ] } keyUpdates:0  reslen:55 1310ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;The length of the pause does seem to be related to the number of connections with more connections causing longer pauses but I need to collect more data to be sure.&lt;/p&gt;

&lt;p&gt;I was originally seeing this problem with our application stress test but have not got YCSB exhibiting the pauses regularly so I can also test any log levels or candidates you may have.&lt;/p&gt;

&lt;p&gt;Rob.&lt;/p&gt;
</comment>
                            <comment id="791610" author="asya" created="Sun, 21 Dec 2014 04:18:51 +0000"  >&lt;p&gt;John, how many cores do you have relative to 128 threads?  If you lower the number of threads to be closer to the number of cores (say no more than double the number of cores) how does that affect the worst latency?&lt;/p&gt;</comment>
                            <comment id="790809" author="john.page" created="Fri, 19 Dec 2014 15:22:24 +0000"  >&lt;p&gt;If you mean my workload the WT cursor is hidden from the mongo API as far&lt;br/&gt;
as I can see, anyone is welcome to review that code.&lt;/p&gt;
</comment>
                            <comment id="788114" author="john.page" created="Wed, 17 Dec 2014 12:02:54 +0000"  >&lt;p&gt;Let me know as soon as you have a candidate fix and I can test for you.&lt;/p&gt;</comment>
                            <comment id="787165" author="john.page" created="Tue, 16 Dec 2014 17:12:48 +0000"  >&lt;p&gt;The code for this is available - it&apos;s very simple - 128 processes on another box  doing 16% insert, 34% update, 50% read - all on primary key. No TTL collections at all.&lt;/p&gt;

&lt;p&gt;Records are 30 fields, approx 1K&lt;/p&gt;
</comment>
                            <comment id="787111" author="john.page" created="Tue, 16 Dec 2014 16:37:46 +0000"  >&lt;p&gt;I&apos;ve tried nojournal to disbale the write ahaead log&lt;br/&gt;
Iv&apos;e tried checkpoint=(wait=2)&lt;br/&gt;
Ive tried slowms=2000 (IN case it&apos;s that log that&apos;s blocking)&lt;/p&gt;

&lt;p&gt;Still seeing 15second pause for update and query at 180 seconds in and 50 second pause at 360 seconds in&lt;/p&gt;</comment>
                            <comment id="787049" author="john.page" created="Tue, 16 Dec 2014 16:02:10 +0000"  >&lt;p&gt;Uploaded as requested.&lt;/p&gt;</comment>
                            <comment id="787042" author="john.page" created="Tue, 16 Dec 2014 15:59:02 +0000"  >&lt;p&gt;Gzipping now - there is 1GB of it.&lt;/p&gt;</comment>
                            <comment id="787034" author="john.page" created="Tue, 16 Dec 2014 15:52:56 +0000"  >&lt;p&gt;Tried nojournal to turn of WAL in WT - it&apos;s not that.&lt;/p&gt;</comment>
                            <comment id="787029" author="john.page" created="Tue, 16 Dec 2014 15:50:16 +0000"  >&lt;p&gt;Ah -except logs can you give me a time range from the files you have and I think I will have the log.&lt;/p&gt;</comment>
                            <comment id="787005" author="john.page" created="Tue, 16 Dec 2014 15:31:38 +0000"  >&lt;p&gt;I have many more graphs and stats I can share  if you want them &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bruce.lucas%4010gen.com&quot; class=&quot;user-hover&quot; rel=&quot;bruce.lucas@10gen.com&quot;&gt;bruce.lucas@10gen.com&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="786881" author="john.page" created="Tue, 16 Dec 2014 11:59:53 +0000"  >&lt;p&gt;Logs uploaded via ssh.&lt;/p&gt;</comment>
                            <comment id="786867" author="john.page" created="Tue, 16 Dec 2014 10:58:04 +0000"  >&lt;p&gt;30 minute run, 128 threads at client end, seeing Queries and Updates blocking for over a minute.&lt;/p&gt;</comment>
                            <comment id="781773" author="john.page" created="Tue, 9 Dec 2014 14:52:28 +0000"  >&lt;p&gt;Also - this is default parameters, 2.8.0-RC-2&lt;/p&gt;</comment>
                            <comment id="779554" author="john.page" created="Fri, 5 Dec 2014 16:17:46 +0000"  >&lt;p&gt;I&apos;m doing some much more extensive testing and rechecking/ruling out any effects at my end - will report back once I have some more solid info.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="173504">SERVER-16479</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="173724">SERVER-16492</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="174846">SERVER-16566</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="175030">SERVER-16572</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="175416">SERVER-16602</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="175936">SERVER-16643</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="175637">SERVER-16619</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="59887" name="WiredTiger.csv.max.png" size="351036" author="john.page@mongodb.com" created="Wed, 17 Dec 2014 16:07:49 +0000"/>
                            <attachment id="59791" name="gdbprof.html" size="7271274" author="bruce.lucas@mongodb.com" created="Tue, 16 Dec 2014 17:45:16 +0000"/>
                            <attachment id="59781" name="incident1-blocker.png" size="328454" author="bruce.lucas@mongodb.com" created="Tue, 16 Dec 2014 16:57:22 +0000"/>
                            <attachment id="59782" name="incident1-waiters-1.png" size="99929" author="bruce.lucas@mongodb.com" created="Tue, 16 Dec 2014 16:57:22 +0000"/>
                            <attachment id="59783" name="incident1-waiters-2.png" size="95491" author="bruce.lucas@mongodb.com" created="Tue, 16 Dec 2014 16:57:22 +0000"/>
                            <attachment id="59792" name="incident1.html" size="956123" author="bruce.lucas@mongodb.com" created="Tue, 16 Dec 2014 17:45:16 +0000"/>
                            <attachment id="59784" name="incident1.png" size="345000" author="bruce.lucas@mongodb.com" created="Tue, 16 Dec 2014 16:57:22 +0000"/>
                            <attachment id="59793" name="incident2.html" size="986761" author="bruce.lucas@mongodb.com" created="Tue, 16 Dec 2014 17:45:16 +0000"/>
                            <attachment id="59750" name="isse.html" size="141304238" author="john.page@mongodb.com" created="Tue, 16 Dec 2014 14:09:10 +0000"/>
                            <attachment id="59888" name="stats.WT.nocomp.csv.max.png" size="594175" author="john.page@mongodb.com" created="Wed, 17 Dec 2014 16:07:50 +0000"/>
                            <attachment id="59817" name="try13.png" size="314225" author="bruce.lucas@mongodb.com" created="Tue, 16 Dec 2014 20:15:57 +0000"/>
                            <attachment id="59818" name="try14.png" size="244019" author="bruce.lucas@mongodb.com" created="Tue, 16 Dec 2014 20:15:57 +0000"/>
                            <attachment id="60411" name="try15.png" size="172673" author="bruce.lucas@mongodb.com" created="Wed, 24 Dec 2014 00:11:27 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>20.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 5 Dec 2014 16:14:51 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        9 years, 7 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            9 years, 7 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>asya.kamsky@mongodb.com</customfieldvalue>
            <customfieldvalue>dan@mongodb.com</customfieldvalue>
            <customfieldvalue>john.page@mongodb.com</customfieldvalue>
            <customfieldvalue>keith.bostic@mongodb.com</customfieldvalue>
            <customfieldvalue>michael.cahill@mongodb.com</customfieldvalue>
            <customfieldvalue>robert.j.moore@allanbank.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrli73:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hs46rb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>149359</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[bruce.lucas@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrkzjj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>