<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:41:51 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-16667] Isolated writes with extreme latencies in replica set under WiredTiger</title>
                <link>https://jira.mongodb.org/browse/SERVER-16667</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;ul&gt;
	&lt;li&gt;tests run on Ubuntu, VMWare, 6 cores.&lt;/li&gt;
	&lt;li&gt;heavy mixed workload (read, update, insert) with 100 clients&lt;/li&gt;
	&lt;li&gt;using a recent pre-rc4 build from master (6105f06402fe1e7578d41f3e4e583a1476ef2455); subsequently reproduced under rc4.&lt;/li&gt;
&lt;/ul&gt;



&lt;p&gt;Run with replica set of 1:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/60433/60433_wt-pre4-jp-100-rs1.png&quot; width=&quot;900&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;Throughout the run mongod write queues (second line) sporadically build, indicating a write bottleneck in mongod. This may be the same issue as &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-16651&quot; title=&quot;Mass deletions and updates (including TTL) completely starve insertions in replica set under WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-16651&quot;&gt;&lt;del&gt;SERVER-16651&lt;/del&gt;&lt;/a&gt;.&lt;/li&gt;
	&lt;li&gt;At a couple points during the run (A, B, C), mongod log reports isolated (single) write operations (updates or inserts) taking extreme times, up to 111891 ms in this test (you can read this off of the &quot;max logged query&quot; graph: vertical axis is longest operation, in ms, logged in each 0.1 second interval). Unknown if this is the same bottleneck or a different issue, so opening this issue to investigate.&lt;/li&gt;
	&lt;li&gt;The pattern seen in the graph above of 3 or so such isolated extreme latencies of increasing size throughout the run is repeatable with this workload. The length of each one is roughly double the preceding one.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;For comparison, here&apos;s the same workload standalone:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/60432/60432_wt-pre4-jp-100-rs0.png&quot; width=&quot;900&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;Write queues do not build.&lt;/li&gt;
	&lt;li&gt;We see pauses in WT (e.g. A, B; subject of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-16662&quot; title=&quot;Extended pauses in WiredTiger when workers greatly exceeds number of cores&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-16662&quot;&gt;&lt;del&gt;SERVER-16662&lt;/del&gt;&lt;/a&gt;). Those pauses are less evident in the replica set case above, probably because the mongod bottleneck in that case throttles the workload going into WT.&lt;/li&gt;
	&lt;li&gt;Associated with pauses are numerous long latency operations around 11 seconds, but no extreme long-latency operations as seen in the preceding test.&lt;/li&gt;
	&lt;li&gt;Overall throughput is much higher in the standalone case, possibly because of the bottleneck in the replica set case.&lt;/li&gt;
&lt;/ul&gt;



&lt;p&gt;Here are the specific queries, two updates and one insert, that showed extreme latencies:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2014-12-24T12:17:14.182-0500 I QUERY    [conn95] command testsrv.$cmd command: update { update: &quot;data&quot;, writeConcern: {}, ordered: true, updates: [ { q: { _id: { o: ObjectId(&apos;549af4f61d41c883ae5622d2&apos;), t: &quot;This is a string, not a small string either, it&apos;s in here to make the index larger&quot;, seq: -1729382253017956352 } }, u: { $set: { f0: -1097749151, f1: -1097691211, f2: &quot;bkfiv   Wfi vWfiv Wf ivW fivW fivW  fiv Wfiv Wf i &quot;, f3: &quot;dsz   szs zsz szs zs zszszs z szsz  szszs z szs zs&quot;, f4: &quot;xa jmH szsz sz sz sz sz szszs zszsz sz  s zszszs z&quot;, f5: &quot;szsz  szs zsz szs zs zszszs z szsz  szszs z szs zs&quot;, f6: &quot;ivWfi  vWfi vWf ivW fiv  Wf i vW f  ivW f ivWfiv W&quot;, f7: &quot;yvWfiv Wfiv Wf iv Wf iv WfivW fivWf iv  W fivWfi v&quot;, f8: &quot;mH s   zszs zsz szs zsz  sz s zs z  szs z szszsz s&quot;, f9: &quot;rqjmH   szs zszsz sz szs zszs zszs  zsz szsz sz s &quot;, fa: 47242, fb: 91501, fc: &quot;ajmH s zszs zs zszsz sz  szsz szsz  szszs zszsz sz&quot;, fd: &quot;mHszsz szsz sz sz sz sz szszs zszsz sz  s zszszs z&quot;, fe: &quot;uHsz  szs zsz szs zs zszszs z szsz  szszs z szs zs&quot;, ff: &quot;zszsz   szs zszsz sz szs zszs zszs  zsz szsz sz s &quot;, f10: &quot;kfivW f i v Wf ivWfi vW fi  v Wfiv  WfivW f ivW fi&quot;, f11: &quot;tkfiv Wfi v W f i vWfiv Wf iv WfivW fivWf ivWfi vW&quot;, f12: &quot;zszs   zszs zsz szs zsz  sz s zs z  szs z szszsz s&quot;, f13: &quot;pKNkf   ivW fivWf iv Wfi vWfi vWfi  vWf ivWf iv W &quot;, f14: 51733, f15: 90312, f16: &quot;qjmHs zs zszs zsz s zsz szs zsz szs zszsz sz szsz &quot;, f17: &quot;yv W   fivW fiv Wfi vWf  iv W fi v  Wfi v WfivWf i&quot;, f18: &quot;bkf i vWfiv Wf ivWf ivW fiv W f iv  Wfi v WfivW fi&quot;, f19: &quot;ajmHs zs zszs zsz s zsz szs zsz szs zszsz sz szsz &quot;, f1a: &quot;bkfivW fivW fiv Wfi vWf  iv W fi v  Wfi v WfivWf i&quot;, f1b: &quot;cnajmH  szs zszsz sz szs zszs zszs  zsz szsz sz s &quot;, f1c: &quot;cn a   jmHs zsz szs zsz  sz s zs z  szs z szszsz s&quot;, f1d: &quot;gt k   fivW fiv Wfi vWf  iv W fi v  Wfi v WfivWf i&quot; } }, multi: false, upsert: false } ] } ntoreturn:1 keyUpdates:0  reslen:95 24494ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2014-12-24T12:18:06.958-0500 I QUERY    [conn100] command testsrv.$cmd command: insert { insert: &quot;data&quot;, writeConcern: {}, ordered: true, documents: [ { _id: { o: ObjectId(&apos;549af4f61d41c883a908b7c2&apos;), t: &quot;This is a string, not a small string either, it&apos;s in here to make the index larger&quot;, seq: 3819052484899373056 }, f0: 1826216753, f1: 1826277117, f2: &quot;bkfivW fivW fiv Wfi vWf  iv W fi v  Wfi v WfivWf i&quot;, f3: &quot;lKNkf iv  W f ivW f ivWf ivWf ivWf ivWfiv WfivW fi&quot;, f4: &quot;wNkfi  vWfi vWf ivW fiv  Wf i vW f  ivW f ivWfiv W&quot;, f5: &quot;qj m   Hszs zsz szs zsz  sz s zs z  szs z szszsz s&quot;, f6: &quot;xajmH   szs zszsz sz szs zszs zszs  zsz szsz sz s &quot;, f7: &quot;oHszs zs zszs zsz s zsz szs zsz szs zszsz sz szsz &quot;, f8: &quot;zsz   szs zsz szs zs zszszs z szsz  szszs z szs zs&quot;, f9: &quot;sz s   zszs zsz szs zsz  sz s zs z  szs z szszsz s&quot;, fa: 54913, fb: 94391, fc: &quot;szsz  szs zsz szs zs zszszs z szsz  szszs z szs zs&quot;, fd: &quot;ivWfi vW fivW fiv W fiv Wfi vWf ivW fivWf iv Wfiv &quot;, fe: &quot;jmHszs zszs zsz szs zsz  sz s zs z  szs z szszsz s&quot;, ff: &quot;fivWf iv  W f ivW f ivWf ivWf ivWf ivWfiv WfivW fi&quot;, f10: &quot;rq jmH szsz sz sz sz sz szszs zszsz sz  s zszszs z&quot;, f11: &quot;wNkfi vW fivW fiv W fiv Wfi vWf ivW fivWf iv Wfiv &quot;, f12: &quot;lKN k fivWf iv Wfiv Wfi vWf i v Wf  ivW f ivWfi vW&quot;, f13: &quot;vWf i vWfiv Wf ivWf ivW fiv W f iv  Wfi v WfivW fi&quot;, f14: 50071, f15: 107864, f16: &quot;mHsz    szs zszsz sz szs zszs zszs  zsz szsz sz s &quot;, f17: &quot;ezszsz szsz sz sz sz sz szszs zszsz sz  s zszszs z&quot;, f18: &quot;yvWfiv Wfiv Wf iv Wf iv WfivW fivWf iv  W fivWfi v&quot;, f19: &quot;tkfiv Wf  i v Wfi v Wfiv Wfiv Wfiv WfivWf ivWfi vW&quot;, f1a: &quot;ivWfi  vWfi vWf ivW fiv  Wf i vW f  ivW f ivWfiv W&quot;, f1b: &quot;fivWf   ivW fivWf iv Wfi vWfi vWfi  vWf ivWf iv W &quot;, f1c: &quot;mHsz    szs zszsz sz szs zszs zszs  zsz szsz sz s &quot;, f1d: &quot;oHsz    szs zszsz sz szs zszs zszs  zsz szsz sz s &quot; } ] } ntoreturn:1 keyUpdates:0  reslen:80 52917ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2014-12-24T12:19:58.656-0500 I QUERY    [conn39] command testsrv.$cmd command: update { update: &quot;data&quot;, writeConcern: {}, ordered: true, updates: [ { q: { _id: { o: ObjectId(&apos;549af4f61d41c8837743de22&apos;), t: &quot;This is a string, not a small string either, it&apos;s in here to make the index larger&quot;, seq: 3602879702735257600 } }, u: { $set: { f0: -1097747785, f1: -1097704122, f2: &quot;rqj m Hszsz sz szsz szs zsz s z sz  szs z szszs zs&quot;, f3: &quot;mHszs zszsz sz szsz szs zsz s z sz  szs z szszs zs&quot;, f4: &quot;bkfivW fivW fiv Wfi vWf  iv W fi v  Wfi v WfivWf i&quot;, f5: &quot;vWfiv   Wfi vWfiv Wf ivW fivW fivW  fiv Wfiv Wf i &quot;, f6: &quot;fivW   fivW fiv Wfi vWf  iv W fi v  Wfi v WfivWf i&quot;, f7: &quot;xajmH sz  s z szs z szsz szsz szsz szszsz szszs zs&quot;, f8: &quot;jmH   szs zsz szs zs zszszs z szsz  szszs z szs zs&quot;, f9: &quot;ajmH s zszs zs zszsz sz  szsz szsz  szszs zszsz sz&quot;, fa: 44944, fb: 82734, fc: &quot;qjmHs zs zszs zsz s zsz szs zsz szs zszsz sz szsz &quot;, fd: &quot;uH s   zszs zsz szs zsz  sz s zs z  szs z szszsz s&quot;, fe: &quot;mHszsz szsz sz sz sz sz szszs zszsz sz  s zszszs z&quot;, ff: &quot;sz s   zszs zsz szs zsz  sz s zs z  szs z szszsz s&quot;, f10: &quot;dszszs zszs zs zszsz sz  szsz szsz  szszs zszsz sz&quot;, f11: &quot;tkfiv   Wfi vWfiv Wf ivW fivW fivW  fiv Wfiv Wf i &quot;, f12: &quot;ezszs zszsz sz szsz szs zsz s z sz  szs z szszs zs&quot;, f13: &quot;ivWf  ivW fiv Wfi vW fivWfi v Wfiv  WfivW f ivW fi&quot;, f14: 40990, f15: 88315, f16: &quot;oHsz    szs zszsz sz szs zszs zszs  zsz szsz sz s &quot;, f17: &quot;ivWfi vW fivW fiv W fiv Wfi vWf ivW fivWf iv Wfiv &quot;, f18: &quot;xa jmH szsz sz sz sz sz szszs zszsz sz  s zszszs z&quot;, f19: &quot;iv W   fivW fiv Wfi vWf  iv W fi v  Wfi v WfivWf i&quot;, f1a: &quot;fivW   fivW fiv Wfi vWf  iv W fi v  Wfi v WfivWf i&quot;, f1b: &quot;bk fiv Wfiv Wf iv Wf iv WfivW fivWf iv  W fivWfi v&quot;, f1c: &quot;kfivWf ivWf iv Wf iv Wf ivWfi vWfiv Wf  i vWfivW f&quot;, f1d: &quot;bkfi   vWfi vWf ivW fiv  Wf i vW f  ivW f ivWfiv W&quot; } }, multi: false, upsert: false } ] } ntoreturn:1 keyUpdates:0  reslen:95 111891ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</description>
                <environment></environment>
        <key id="176091">SERVER-16667</key>
            <summary>Isolated writes with extreme latencies in replica set under WiredTiger</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="michael.cahill@mongodb.com">Michael Cahill</assignee>
                                    <reporter username="bruce.lucas@mongodb.com">Bruce Lucas</reporter>
                        <labels>
                            <label>wiredtiger</label>
                    </labels>
                <created>Wed, 24 Dec 2014 18:17:46 +0000</created>
                <updated>Sun, 18 Jan 2015 16:35:23 +0000</updated>
                            <resolved>Tue, 6 Jan 2015 03:56:26 +0000</resolved>
                                    <version>2.8.0-rc4</version>
                                    <fixVersion>2.8.0-rc5</fixVersion>
                                    <component>Concurrency</component>
                                        <votes>0</votes>
                                    <watches>22</watches>
                                                                                                                <comments>
                            <comment id="797793" author="michael.cahill" created="Tue, 6 Jan 2015 03:56:26 +0000"  >&lt;p&gt;The issues about scaling with many more threads than cores are captured in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-16662&quot; title=&quot;Extended pauses in WiredTiger when workers greatly exceeds number of cores&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-16662&quot;&gt;&lt;del&gt;SERVER-16662&lt;/del&gt;&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="796413" author="xgen-internal-githook" created="Fri, 2 Jan 2015 23:32:54 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;michaelcahill&apos;, u&apos;name&apos;: u&apos;Michael Cahill&apos;, u&apos;email&apos;: u&apos;michael.cahill@wiredtiger.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-16667&quot; title=&quot;Isolated writes with extreme latencies in replica set under WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-16667&quot;&gt;&lt;del&gt;SERVER-16667&lt;/del&gt;&lt;/a&gt; Fix WT truncate of all records in a capped collection.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/53d7bce974d18fc4a92262de5ac6e604b79d8a04&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/53d7bce974d18fc4a92262de5ac6e604b79d8a04&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="796092" author="xgen-internal-githook" created="Fri, 2 Jan 2015 05:24:20 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;michaelcahill&apos;, u&apos;name&apos;: u&apos;Michael Cahill&apos;, u&apos;email&apos;: u&apos;michael.cahill@wiredtiger.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-16667&quot; title=&quot;Isolated writes with extreme latencies in replica set under WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-16667&quot;&gt;&lt;del&gt;SERVER-16667&lt;/del&gt;&lt;/a&gt; Only truncate if we find some documents.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/2f6817253ebfef52f06c55f5a58a799997e014ad&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/2f6817253ebfef52f06c55f5a58a799997e014ad&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="796089" author="xgen-internal-githook" created="Fri, 2 Jan 2015 04:33:26 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;michaelcahill&apos;, u&apos;name&apos;: u&apos;Michael Cahill&apos;, u&apos;email&apos;: u&apos;michael.cahill@wiredtiger.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-16667&quot; title=&quot;Isolated writes with extreme latencies in replica set under WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-16667&quot;&gt;&lt;del&gt;SERVER-16667&lt;/del&gt;&lt;/a&gt; Faster capped collections in WT.&lt;/p&gt;

&lt;p&gt;Use a single WiredTiger truncate operation rather than multiple delete&lt;br/&gt;
operations to delete from a capped collection.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/1b522cf5745c40754f9d5aa4f4e35def4e8fee74&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/1b522cf5745c40754f9d5aa4f4e35def4e8fee74&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="794444" author="xgen-internal-githook" created="Mon, 29 Dec 2014 05:31:49 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;michaelcahill&apos;, u&apos;name&apos;: u&apos;Michael Cahill&apos;, u&apos;email&apos;: u&apos;michael.cahill@wiredtiger.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-16667&quot; title=&quot;Isolated writes with extreme latencies in replica set under WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-16667&quot;&gt;&lt;del&gt;SERVER-16667&lt;/del&gt;&lt;/a&gt; Have threads block if they insert enough additional records while a capped collection is being truncated.&lt;br/&gt;
Branch: capped-truncate&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/e15538f67138812d647d047efea4c3fb5e5023de&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/e15538f67138812d647d047efea4c3fb5e5023de&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="794443" author="xgen-internal-githook" created="Mon, 29 Dec 2014 05:31:48 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;michaelcahill&apos;, u&apos;name&apos;: u&apos;Michael Cahill&apos;, u&apos;email&apos;: u&apos;michael.cahill@wiredtiger.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-16667&quot; title=&quot;Isolated writes with extreme latencies in replica set under WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-16667&quot;&gt;&lt;del&gt;SERVER-16667&lt;/del&gt;&lt;/a&gt; Use a single truncate operation to manage capped collections rather than multiple deletes.&lt;/p&gt;

&lt;p&gt;Bound the work done managing capped collections for any individual update to avoid long latencies if deletes can&apos;t keep up with inserts.&lt;br/&gt;
Branch: capped-truncate&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/79a784b2809ad1ed7ce5be361f66f86d8e7049ea&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/79a784b2809ad1ed7ce5be361f66f86d8e7049ea&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="794055" author="bruce.lucas@10gen.com" created="Fri, 26 Dec 2014 15:06:07 +0000"  >&lt;p&gt;Looks like the isolated ops with extreme latencies are stuck inside WT, not the mongod layer.&lt;/p&gt;

&lt;p&gt;Here&apos;s a long op that ran for 78 seconds on conn76, from about 09:34:43 to 09:36:01:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2014-12-26T09:36:01.448-0500 I QUERY    [conn76] command testsrv.$cmd command: update { update: &quot;data&quot;, writeConcern: {}, ordered: true, updates: [ { q: { _id: { o: ObjectId(&apos;549d718a1d41c848573ac682&apos;), t: &quot;This is a string, not a small string either, it&apos;s in here to make the index larger&quot;, seq: -1297036684394758144 } }, u: { $set: { f0: -295740523, f1: -295695630, f2: &quot;dszsz szs zsz szs zs zszszs z szsz  szszs z szs zs&quot;, f3: &quot;jmHsz sz  s z szs z szsz szsz szsz szszsz szszs zs&quot;, f4: &quot;hgt   kfi vWf ivW fi vWfivW f ivWf  ivWfi v Wfi vW&quot;, f5: &quot;uHsz s zszs zs zszsz sz  szsz szsz  szszs zszsz sz&quot;, f6: &quot;rqjmH szs zsz szs zs zszszs z szsz  szszs z szs zs&quot;, f7: &quot;qj m   Hszs zsz szs zsz  sz s zs z  szs z szszsz s&quot;, f8: &quot;kfivW f i v Wf ivWfi vW fi  v Wfiv  WfivW f ivW fi&quot;, f9: &quot;wNkfi v W f iv WfivW fi vW  f ivWf  ivWfi v Wfi vW&quot;, fa: 38995, fb: 103702, fc: &quot;ds zsz szsz sz sz sz sz szszs zszsz sz  s zszszs z&quot;, fd: &quot;uHszs  zszs zsz szs zsz  sz s zs z  szs z szszsz s&quot;, fe: &quot;jm Hsz szsz sz sz sz sz szszs zszsz sz  s zszszs z&quot;, ff: &quot;kfivWf ivWf iv Wf iv Wf ivWfi vWfiv Wf  i vWfivW f&quot;, f10: &quot;gtkfiv Wfiv Wf iv Wf iv WfivW fivWf iv  W fivWfi v&quot;, f11: &quot;zsz   szs zsz szs zs zszszs z szsz  szszs z szs zs&quot;, f12: &quot;ezszs  zszs zsz szs zsz  sz s zs z  szs z szszsz s&quot;, f13: &quot;uHsz s zszs zs zszsz sz  szsz szsz  szszs zszsz sz&quot;, f14: 52194, f15: 104076, f16: &quot;fivWf   ivW fivWf iv Wfi vWfi vWfi  vWf ivWf iv W &quot;, f17: &quot;mHsz  szs zsz szs zs zszszs z szsz  szszs z szs zs&quot;, f18: &quot;naj   mHs zsz szs zs zszszs z szsz  szszs z szs zs&quot;, f19: &quot;gtkfiv Wfiv Wf iv Wf iv WfivW fivWf iv  W fivWfi v&quot;, f1a: &quot;mH s   zszs zsz szs zsz  sz s zs z  szs z szszsz s&quot;, f1b: &quot;hgtk   fivW fiv Wfi vWf  iv W fi v  Wfi v WfivWf i&quot;, f1c: &quot;oHsz  szs zsz szs zs zszszs z szsz  szszs z szs zs&quot;, f1d: &quot;najmH szs z s z s zszsz sz sz szszs zszsz szszs zs&quot; } }, multi: false, upsert: false } ] } ntoreturn:1 keyUpdates:0  reslen:95 78144ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;I had enchanced the &quot;end connection&quot; message to include the tid at the end of the message, which is 18578 in this case:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2014-12-26T09:36:01.449-0500 I NETWORK  [conn76] end connection 127.0.0.1:33528 (1 connection now open) 18578&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;The above run was done with gdb sampling every 5 seconds throughout the run; focusing on the call tree for those stack samples for just tid (LWP) 18578, we see that it&apos;s spending its time during that long running op (marked with blue lines below) doing sched_yield in one of three places inside WT, all within __wt_log_write:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/60479/60479_profile.png&quot; width=&quot;600&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;Also attaching the raw stack trace samples from gdb as gdbmon.log from which the above call tree was generated, and we can double-check there that LWP 18578 is seen in __wt_log_write calling sched_yield in every sample from 09:34:46.342897 onwards, corresponding to the time that the op was stuck.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="178969">SERVER-16868</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="175994">SERVER-16651</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="175869">SERVER-16640</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="60516" name="capped-truncate-latencies.png" size="577971" author="michael.cahill@mongodb.com" created="Mon, 29 Dec 2014 06:46:24 +0000"/>
                            <attachment id="60480" name="gdbmon.log" size="10270552" author="bruce.lucas@mongodb.com" created="Fri, 26 Dec 2014 15:02:23 +0000"/>
                            <attachment id="60510" name="oplog-inserts-overtake-deletes.png" size="103450" author="bruce.lucas@mongodb.com" created="Sun, 28 Dec 2014 18:00:48 +0000"/>
                            <attachment id="60479" name="profile.png" size="384407" author="bruce.lucas@mongodb.com" created="Fri, 26 Dec 2014 15:02:23 +0000"/>
                            <attachment id="60432" name="wt-pre4-jp-100-rs0.png" size="127268" author="bruce.lucas@mongodb.com" created="Wed, 24 Dec 2014 18:17:46 +0000"/>
                            <attachment id="60433" name="wt-pre4-jp-100-rs1.png" size="167989" author="bruce.lucas@mongodb.com" created="Wed, 24 Dec 2014 18:17:46 +0000"/>
                            <attachment id="60501" name="wt-schwerin-jp-100-rs1-fix.png" size="89192" author="bruce.lucas@mongodb.com" created="Sat, 27 Dec 2014 04:10:56 +0000"/>
                            <attachment id="60502" name="wt-schwerin-jp-100-rs1-nofix.png" size="95592" author="bruce.lucas@mongodb.com" created="Sat, 27 Dec 2014 04:10:56 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 25 Dec 2014 03:02:38 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        9 years, 6 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            9 years, 6 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>michael.cahill@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrlfyf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hs51br:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>154382</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsgdan:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>