<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:05:21 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-24094] Server cache use can take up to an hour to recover from heavy load </title>
                <link>https://jira.mongodb.org/browse/SERVER-24094</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We have found that the last 3 releases of mongod can get into a state  whereby the server is unable to service read tickets in anything like a timely manner. We find this occurring when the primary shard is under heavy read and write load. Once the server is in this state (%cache used ~=96%, %cache dirty ~&amp;gt; 30,  read tickets available ~&amp;gt;60) for more than ~15 minutes it will remain in that state for up to an hour even when query load is significantly reduced (reduced to ~1/3 of previous load).&lt;/p&gt;

&lt;p&gt;Checking tps and cpu load with iostat I see that the tps load is very low (usually under 1000 on ephemeral SSD storage) but cpu load is very high (usually 85% with system use taking anywhere between 15 and 50% of cpu load).&lt;/p&gt;

&lt;p&gt;Looking in the logs for long-running queries or heavy query loads when in this condition shows no document scans. Checking db.currentOp() can return over 200 operations outstanding but no operations with secs_running&amp;gt;1. &lt;/p&gt;

&lt;p&gt;We are running a 3.2.6 sharded cluster on ec2. The ami id is amzn-ami-hvm-2015.03.0.x86_64-gp2 (ami-1ecae776). Instance type is i2.xLarge. No other services are running on the server.&lt;/p&gt;</description>
                <environment></environment>
        <key id="285344">SERVER-24094</key>
            <summary>Server cache use can take up to an hour to recover from heavy load </summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="ramon.fernandez@mongodb.com">Ramon Fernandez Marina</assignee>
                                    <reporter username="miketempleman">Mike Templeman</reporter>
                        <labels>
                    </labels>
                <created>Sun, 8 May 2016 18:11:32 +0000</created>
                <updated>Fri, 22 Jul 2016 18:28:35 +0000</updated>
                            <resolved>Fri, 22 Jul 2016 18:28:35 +0000</resolved>
                                    <version>3.2.4</version>
                    <version>3.2.5</version>
                    <version>3.2.6</version>
                                                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>9</watches>
                                                                                                                <comments>
                            <comment id="1332957" author="thomas.schubert" created="Fri, 22 Jul 2016 18:28:21 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=miketempleman&quot; class=&quot;user-hover&quot; rel=&quot;miketempleman&quot;&gt;miketempleman&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=gheppner&quot; class=&quot;user-hover&quot; rel=&quot;gheppner&quot;&gt;gheppner&lt;/a&gt; has seen &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-24580?focusedCommentId=1325474&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-1325474&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;encouraging results&lt;/a&gt; after upgrading to 3.2.8. Therefore, I am closing this ticket as a duplicate of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-24580&quot; title=&quot;Improve performance when WiredTiger cache is full&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-24580&quot;&gt;&lt;del&gt;SERVER-24580&lt;/del&gt;&lt;/a&gt;. Please upgrade to 3.2.8 and confirm that the issue is resolved. &lt;/p&gt;

&lt;p&gt;If you still experience issues after upgrading, please let us know and we will continue to investigate.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1309204" author="ramon.fernandez" created="Tue, 28 Jun 2016 23:23:03 +0000"  >&lt;p&gt;When I wrote &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-24094?focusedCommentId=1308163&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-1308163&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;the comment above&lt;/a&gt; the changes in the provided patch build had only made it to the development branch and into the 3.3.9 development release (announced earlier today).&lt;/p&gt;

&lt;p&gt;As of today, &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-24580?focusedCommentId=1309006&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-1309006&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;these changes have also been added to the v3.2 branch&lt;/a&gt;, and the goal is to make them part of the next stable release, 3.2.8, scheduled for the coming weeks.&lt;/p&gt;

&lt;p&gt;Hope that clarifies things, but please let us know if you have additional questions.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="1309158" author="gheppner" created="Tue, 28 Jun 2016 21:53:37 +0000"  >&lt;p&gt;Thanks. I&apos;ll keep you posted. As I mentioned, we see this less frequently in our dev environment so may be a while before I have any conclusive results.  And I too am somewhat hesitant to put something like this out in production, although might consider it if we don&apos;t see any issues running it in Dev.&lt;/p&gt;

&lt;p&gt;Can you clarify for me when the changes in this patch would see a production release? You mentioned a 3.3.x development release, so does that mean it won&apos;t be until 3.4.x?&lt;/p&gt;</comment>
                            <comment id="1309116" author="ramon.fernandez" created="Tue, 28 Jun 2016 20:57:27 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=gheppner&quot; class=&quot;user-hover&quot; rel=&quot;gheppner&quot;&gt;gheppner&lt;/a&gt;, I&apos;ve attached a file to this ticket with a patched version of MongoDB for Ubuntu 14.04 LTS that you can use for testing:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;mongo-mongodb_mongo_v3.2_ubuntu1404_30162fa8bbb9d7e7f7a789361aed7e046995f7b3_16_06_28_18_55_58.tgz&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Please let us know how it goes.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="1308996" author="gheppner" created="Tue, 28 Jun 2016 18:56:24 +0000"  >&lt;p&gt;We&apos;re on Ubuntu 14.04 LTS.&lt;/p&gt;</comment>
                            <comment id="1308872" author="ramon.fernandez" created="Tue, 28 Jun 2016 17:13:51 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=gheppner&quot; class=&quot;user-hover&quot; rel=&quot;gheppner&quot;&gt;gheppner&lt;/a&gt;, we have a patch that has passed all our internal qualification tests, so we can send it your way for testing. Can you please tell me what platform(s) will you be testing on so I can produce the right binaries?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="1308163" author="ramon.fernandez" created="Mon, 27 Jun 2016 23:17:38 +0000"  >&lt;p&gt;Thanks for the additional input &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=gheppner&quot; class=&quot;user-hover&quot; rel=&quot;gheppner&quot;&gt;gheppner&lt;/a&gt;. The current changes for &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-24580&quot; title=&quot;Improve performance when WiredTiger cache is full&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-24580&quot;&gt;&lt;del&gt;SERVER-24580&lt;/del&gt;&lt;/a&gt; are part of development release 3.3.9, which will be available soon &amp;#8211; but I&apos;ll work on getting a patch build on MongoDB 3.2 available in case you&apos;re able to find resources to test it.&lt;/p&gt;</comment>
                            <comment id="1308155" author="gheppner" created="Mon, 27 Jun 2016 23:10:58 +0000"  >&lt;p&gt;Per my comment - the node in &quot;blue&quot; is routinely using 100% of its cache despite query volume dropping off, and the other two shards only at 80%. &lt;/p&gt;</comment>
                            <comment id="1308154" author="gheppner" created="Mon, 27 Jun 2016 23:09:43 +0000"  >&lt;p&gt;I wanted to chime in and say we are being impacted by this exact same issue, on Mongo 3.2.5.  In the early versions of 3.2.x we did not see this problem, and first noticed it on 3.2.4.  &lt;/p&gt;

&lt;p&gt;I&apos;m attaching a graph that tells a very similar story to what the reporter is seeing.  In our case, I do (much less frequently) see this same issue in our Dev environment, so testing a patch might be an option for us. This is causing us fairly serious problems, as you can see in the attached image having one shard experience such huge delays (which comes with huge load average and system % time) is badly hurting our throughput.  I am debating scaling up on either shards or memory since I do know our working set is fairly large, but I&apos;m hesitant since if it were not for this issue I feel like we&apos;d be getting adequate performance with what we have today.  &lt;/p&gt;

&lt;p&gt;If providing correlating diagnostic data from our deployment would be of help here I&apos;m happy to try and provide it.  &lt;/p&gt;</comment>
                            <comment id="1303456" author="ramon.fernandez" created="Wed, 22 Jun 2016 20:48:05 +0000"  >&lt;p&gt;No problem, given your current situation I&apos;d do the same thing. Thanks for getting back to us, we&apos;ll keep you posted on any developments.&lt;/p&gt;</comment>
                            <comment id="1303444" author="mike@meshfire.com" created="Wed, 22 Jun 2016 20:38:27 +0000"  >&lt;p&gt;Ramon&lt;/p&gt;

&lt;p&gt;I am understandably reluctant to run a patch on our production system.&lt;br/&gt;
Particularly if the patch has any possibility of data loss or corruption. I&lt;br/&gt;
cannot effectively simulate this problem on our staging system (load,&lt;br/&gt;
system setup, etc.) either. Given that the problem occurs on our primary&lt;br/&gt;
shard, any potential data loss of semi-relational collections (yeah, I&lt;br/&gt;
know) would require us to roll back to the last hour of backup and restore.&lt;br/&gt;
Since we have not tested a restore for months (again, yeah, I know) i&lt;br/&gt;
cannot expect 100% data recovery.&lt;/p&gt;

&lt;p&gt;Mike&lt;/p&gt;

&lt;p&gt;&amp;#8211; &lt;/p&gt;

&lt;p&gt;&lt;b&gt;Mike Templeman&lt;/b&gt;&lt;br/&gt;
&lt;b&gt;Head of Development&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;T: @talkingfrog1950 &amp;lt;&lt;a href=&quot;http://twitter.com/missdestructo&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://twitter.com/missdestructo&lt;/a&gt;&amp;gt;&lt;br/&gt;
T: @Meshfire &amp;lt;&lt;a href=&quot;http://twitter.com/meshfire&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://twitter.com/meshfire&lt;/a&gt;&amp;gt;&lt;/p&gt;


&lt;p&gt;On Wed, Jun 22, 2016 at 5:11 AM, Ramon Fernandez (JIRA) &amp;lt;jira@mongodb.org&amp;gt;&lt;/p&gt;
</comment>
                            <comment id="1302478" author="ramon.fernandez" created="Wed, 22 Jun 2016 12:11:20 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=miketempleman&quot; class=&quot;user-hover&quot; rel=&quot;miketempleman&quot;&gt;miketempleman&lt;/a&gt;, after taking a closer look at the diagnostic data you uploaded the evidence for you being affected by &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-24580&quot; title=&quot;Improve performance when WiredTiger cache is full&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-24580&quot;&gt;&lt;del&gt;SERVER-24580&lt;/del&gt;&lt;/a&gt; is inconclusive. We&apos;re working on a fix for it, and I was wondering if you&apos;d have an environment where you could try out a patch build with such tentative fix.&lt;/p&gt;

&lt;p&gt;I understand testing a patch build is not ideal if you&apos;re seeing this in production, but field testing would help us make progress on this ticket.&lt;/p&gt;

&lt;p&gt;Please let us know your thoughts. In the meantime we&apos;re also investigating whether this could be a manifestation of a variation of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-23622&quot; title=&quot;Inconsistent throughput during insert workload&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-23622&quot;&gt;&lt;del&gt;SERVER-23622&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="1301732" author="miketempleman" created="Tue, 21 Jun 2016 19:16:35 +0000"  >&lt;p&gt;Ramon&lt;/p&gt;

&lt;p&gt;No worries about the delay. This clearly is a difficult problem and it might well be the same one as &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-24580&quot; title=&quot;Improve performance when WiredTiger cache is full&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-24580&quot;&gt;&lt;del&gt;SERVER-24580&lt;/del&gt;&lt;/a&gt;. While I do not have the same analysis tools available to your folks I have examined my issue for some months now and have some observations that might help a little:&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;the affected mongod process is not i/o bound at all. I am using ephemeral SSD storage and when in the stall state TPS is usually less than 1,000 (I see TPS burst often to &amp;gt;10,000 without any issues)&lt;/li&gt;
&lt;/ul&gt;


&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Virtually always I see iostat %system go to &amp;gt; 40% during the stall period&lt;/li&gt;
&lt;/ul&gt;


&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;%dirty is generally &amp;gt; 25% during the entire stall period&lt;/li&gt;
&lt;/ul&gt;


&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;%used is invariably = 96%. Very rarely greater than that.&lt;/li&gt;
&lt;/ul&gt;


&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;vsize seems more important than resident size. Default cache size on my servers is 16GB (WT default). When vsize goes to ~24GB on the primary shard primary during the busy daytime I start to schedule my usual workaround&lt;/li&gt;
&lt;/ul&gt;


&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;the number of read tickets available during the stall drops to &amp;lt;30 (average). Write tickets drop to ~100&lt;/li&gt;
&lt;/ul&gt;


&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Workaround is to (1) reduce the number of reads, (2) step down the primary server, and (3) restart the primary server. This often works unless the secondary vsize is &amp;gt; 20GB, in which case the same situation occurs on the new primary within a minute or two.&lt;/li&gt;
&lt;/ul&gt;


&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;balancing is turned off&lt;/li&gt;
&lt;/ul&gt;


&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;fast compression is used.&lt;/li&gt;
&lt;/ul&gt;


&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;OS is amzn-ami-hvm-2015.03.0.x86_64-gp2 (ami-1ecae776)&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;The attached image shows the period of time going from (1) everything is working without a problem, (2) stall occurred on primary shard primary, (3) stepdown of primary followed by restart of affected server.&lt;/p&gt;

&lt;p&gt;Some of the behavior does indicate a possible deadly embrace, though I would not expect CPU load to go so high in that situation. I was thinking that the issue was limited cpu capacity when the cache is fully loaded but after reading &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-24580&quot; title=&quot;Improve performance when WiredTiger cache is full&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-24580&quot;&gt;&lt;del&gt;SERVER-24580&lt;/del&gt;&lt;/a&gt; I am less certain.&lt;/p&gt;

&lt;p&gt;I hope this information provides a little help to the engineers.&lt;/p&gt;

&lt;p&gt;Mike&lt;/p&gt;</comment>
                            <comment id="1301516" author="ramon.fernandez" created="Tue, 21 Jun 2016 17:17:48 +0000"  >&lt;p&gt;Apologies for the radio silence &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=miketempleman&quot; class=&quot;user-hover&quot; rel=&quot;miketempleman&quot;&gt;miketempleman&lt;/a&gt;. This issue has proven to be elusive, but there&apos;s been some recent developments: I believe the symptoms you describe are the same as those in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-24580&quot; title=&quot;Improve performance when WiredTiger cache is full&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-24580&quot;&gt;&lt;del&gt;SERVER-24580&lt;/del&gt;&lt;/a&gt;, which is being investigated at high priority.&lt;/p&gt;

&lt;p&gt;I&apos;ll check with my colleagues, but if they agree that you&apos;re likely running into &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-24580&quot; title=&quot;Improve performance when WiredTiger cache is full&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-24580&quot;&gt;&lt;del&gt;SERVER-24580&lt;/del&gt;&lt;/a&gt; I&apos;ll be closing this ticket as a duplicate of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-24580&quot; title=&quot;Improve performance when WiredTiger cache is full&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-24580&quot;&gt;&lt;del&gt;SERVER-24580&lt;/del&gt;&lt;/a&gt; &amp;#8211; which I suggest you watch for updates.&lt;/p&gt;

&lt;p&gt;Thanks for your patience on this matter.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="1262951" author="miketempleman" created="Thu, 12 May 2016 17:46:12 +0000"  >&lt;p&gt;Could the issue be simply that I need more cpu resources to handle the compression/decompression processing by WT given the load? It doesn&apos;t appear that io latency is an issue.&lt;/p&gt;</comment>
                            <comment id="1261192" author="miketempleman" created="Tue, 10 May 2016 19:58:55 +0000"  >&lt;p&gt;The %system cpu load when in this state seems very high as well. Here is some iostat 5 output:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;avg-cpu:  %user   %nice %system %iowait  %steal   %idle&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;          51.00    0.00   47.49    0.10    0.05    1.35&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;xvda              0.40         0.00         4.80          0         24&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;xvdb           1077.40      6228.80     19368.00      31144      96840&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;avg-cpu:  %user   %nice %system %iowait  %steal   %idle&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;          52.54    0.00   45.00    0.40    0.05    2.01&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;xvda              0.00         0.00         0.00          0          0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;xvdb           3578.40      5400.00     69345.60      27000     346728&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="1258924" author="miketempleman" created="Mon, 9 May 2016 04:16:43 +0000"  >&lt;p&gt;Attached are the diagnostic logs for the last 5 days. The latest log covers a period where the core server/WT entered the cache load/ticket exhaustion state.&lt;/p&gt;</comment>
                            <comment id="1258779" author="dan@10gen.com" created="Sun, 8 May 2016 20:28:43 +0000"  >&lt;p&gt;Hi Mike,&lt;br/&gt;
If possible, can you zip up and attach to this ticket the diagnostic log files found in &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt; for the server?&lt;/p&gt;

&lt;p&gt;Thanks, Dan&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="293974">SERVER-24580</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="120906" name="mongo-5-6-2.log.zip" size="64739468" author="miketempleman" created="Sun, 8 May 2016 18:11:32 +0000"/>
                            <attachment id="128788" name="mongo-mongodb_mongo_v3.2_ubuntu1404_30162fa8bbb9d7e7f7a789361aed7e046995f7b3_16_06_28_18_55_58.tgz" size="84738695" author="ramon.fernandez@mongodb.com" created="Tue, 28 Jun 2016 20:55:52 +0000"/>
                            <attachment id="127795" name="mongo-server-stall" size="187501" author="miketempleman" created="Tue, 21 Jun 2016 19:16:35 +0000"/>
                            <attachment id="120934" name="mongoPrimaryDiagnostics.zip" size="107488453" author="miketempleman" created="Mon, 9 May 2016 04:16:43 +0000"/>
                            <attachment id="128675" name="wired_tiger_cache_full.png" size="666400" author="gheppner" created="Mon, 27 Jun 2016 23:10:58 +0000"/>
                            <attachment id="120907" name="wiredtiger status.txt" size="24248" author="miketempleman" created="Sun, 8 May 2016 18:11:32 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>18.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Sun, 8 May 2016 20:28:43 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        7 years, 29 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            7 years, 29 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>dan@mongodb.com</customfieldvalue>
            <customfieldvalue>gheppner</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
            <customfieldvalue>mike@meshfire.com</customfieldvalue>
            <customfieldvalue>miketempleman</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrk8cf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hsl85b:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;One our system this occurs several times a day during the work week (PST).&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hseypr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>