<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:02:03 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-23001] Occasional 100% cache uses cripples server</title>
                <link>https://jira.mongodb.org/browse/SERVER-23001</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Apologies this may be a WT ticket?&lt;/p&gt;

&lt;p&gt;When running WiredTiger in a 3 node replica set, after a period of time one or more of the nodes will consistently run above 80% cache used and grow until it hits 100% at which point it will struggle to serve reads (or writes when master) in a timely fashion.&lt;/p&gt;

&lt;p&gt;Usually a restart and or stepDown helps.&lt;/p&gt;

&lt;p&gt;So far adding more eviction threads or reducing the eviction trigger and target does not help, nor does providing more RAM/cache.&lt;/p&gt;

&lt;p&gt;Before this happens mongo uses ~45-50% ram, once the issue presents ram goes over 50% - anecdotally sometimes up to 80%.&lt;/p&gt;

&lt;p&gt;The issue seems to occur randomly after days or more of apparent stability.&lt;/p&gt;

&lt;p&gt;There is no apparent stress on the VM. More RAM did not help, more CPU seems to have reduced the rate of ocurrances (was on an m4.2xlarge). Before running on the current instances there was a lot of system CPU.&lt;/p&gt;

&lt;p&gt;Running on 3 x AWS c4.4xlarge, 16 Cores, 60 GB Ram.&lt;/p&gt;

&lt;p&gt;Mongostat output from the affected slave (during the issue):&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;insert query update delete getmore command % dirty % used flushes vsize   res qr|qw ar|aw netIn netOut conn   set repl     time&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;   *50   446  *1720     *3       4   274|0     0.3  100.0       0 17.4G 15.8G   0|0  1|16   85k     5m  681 floow  SEC 09:13:32&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;   *52   323  *3099    *91       1   113|0     0.5  100.0       0 17.4G 15.8G   0|3  27|0   60k     7m  681 floow  SEC 09:13:33&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;   *50   220  *2825     *6       4    75|0     0.2   99.7       0 17.4G 15.8G   0|0   1|2   43k     8m  681 floow  SEC 09:13:34&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;   *35   221  *3417    *11       2   263|0     0.3  100.0       0 17.4G 15.8G   0|2  1|12   54k     2m  681 floow  SEC 09:13:35&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;   *18   432  *2013    *12       8   344|0     0.2  100.0       0 17.4G 15.8G   2|3  19|0   92k    16m  681 floow  SEC 09:13:36&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;IOStat:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;03/08/2016 09:14:09 AM&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;avg-cpu:  %user   %nice %system %iowait  %steal   %idle&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;          13.17    0.00    4.62    2.70    0.13   79.38&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;xvda              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;xvdz              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;xvdb              0.00     0.00 1455.00    0.00 16260.00     0.00    22.35     1.06    0.74    0.74    0.00   0.16  22.80&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;iotop:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Total DISK READ :      13.69 M/s | Total DISK WRITE :      41.78 M/s&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Actual DISK READ:      13.72 M/s | Actual DISK WRITE:      23.58 M/s&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;   TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO&amp;gt;    COMMAND&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; 72806 be/4 mongodb     3.83 K/s    8.72 M/s  0.00 % 54.51 % mongod --config /etc/mongod.conf&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;   686 be/3 root        0.00 B/s  881.69 K/s  0.00 % 14.18 % [jbd2/xvdb-8]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;129706 be/4 mongodb  1153.86 K/s   46.00 K/s  0.00 %  7.27 % mongod --config /etc/mongod.conf&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</description>
                <environment>Distributor ID:	Ubuntu&lt;br/&gt;
Description:	Ubuntu 14.04.4 LTS&lt;br/&gt;
Release:	14.04&lt;br/&gt;
Codename:	trusty&lt;br/&gt;
&lt;br/&gt;
mongod -version&lt;br/&gt;
db version v3.0.9&lt;br/&gt;
git version: 20d60d3491908f1ae252fe452300de3978a040c7&lt;br/&gt;
&lt;br/&gt;
AWS c4.4xlarge&lt;br/&gt;
&lt;br/&gt;
1 x 6TB gp2 volume for mongo exclusively&lt;br/&gt;
&lt;br/&gt;
No other significant processes or services running/consuming resource.</environment>
        <key id="270707">SERVER-23001</key>
            <summary>Occasional 100% cache uses cripples server</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="-1">Unassigned</assignee>
                                    <reporter username="paul.ridgway">Paul Ridgway</reporter>
                        <labels>
                    </labels>
                <created>Tue, 8 Mar 2016 09:25:35 +0000</created>
                <updated>Sat, 18 Jun 2016 11:48:35 +0000</updated>
                            <resolved>Mon, 28 Mar 2016 19:16:08 +0000</resolved>
                                    <version>3.0.9</version>
                                                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>11</watches>
                                                                                                                <comments>
                            <comment id="1298754" author="bruce.lucas@10gen.com" created="Sat, 18 Jun 2016 11:48:12 +0000"  >&lt;p&gt;Thanks Paul. I can confirm that this looks like an instance of the issue or issues we&apos;re investigating on &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-24580&quot; title=&quot;Improve performance when WiredTiger cache is full&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-24580&quot;&gt;&lt;del&gt;SERVER-24580&lt;/del&gt;&lt;/a&gt;: when WT cache utilization reaches 95% application threads get involved in evictions, and we see large gaps in the data collection, confirming that application threads in general are seeing very high latencies.&lt;/p&gt;</comment>
                            <comment id="1298747" author="paul.ridgway" created="Sat, 18 Jun 2016 09:16:56 +0000"  >&lt;p&gt;I&apos;ve got the diagnostics from the 3 nodes. The primary moved around a lot in the chaos and there was lots of restarting and resizing instances (none of which helped). I uploaded them using the Support File Upload tool - hope it helps! &lt;/p&gt;

&lt;p&gt;Thanks&lt;/p&gt;</comment>
                            <comment id="1298196" author="ramon.fernandez" created="Fri, 17 Jun 2016 18:02:19 +0000"  >&lt;p&gt;Thanks for the update &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=paul%40thefloow.com&quot; class=&quot;user-hover&quot; rel=&quot;paul@thefloow.com&quot;&gt;paul@thefloow.com&lt;/a&gt;, sorry to hear 3.2 did not fix your use case. We&apos;re investigating performance issues when the WiredTiger cache fills up in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-24580&quot; title=&quot;Improve performance when WiredTiger cache is full&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-24580&quot;&gt;&lt;del&gt;SERVER-24580&lt;/del&gt;&lt;/a&gt;, feel free to tune to that ticket for further updates.&lt;/p&gt;

&lt;p&gt;MongoDB 3.2 automatically collects diagnostic data in the &lt;tt&gt;diagnostic.data&lt;/tt&gt; directory inside your &lt;tt&gt;dbpath&lt;/tt&gt;. If you still have those files and could upload them to this ticket we could investigate whether this indeed matches &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-24580&quot; title=&quot;Improve performance when WiredTiger cache is full&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-24580&quot;&gt;&lt;del&gt;SERVER-24580&lt;/del&gt;&lt;/a&gt; or is a different problem. You can open a new ticket as well.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="1298139" author="paul.ridgway" created="Fri, 17 Jun 2016 17:40:45 +0000"  >&lt;p&gt;In a high volume prod env 3.2 was catastrophically worse and we had to roll back. We plan to update some slaves to get logs when we can. In testing it was fine but it was disastrous under the production workload ...&lt;/p&gt;

&lt;p&gt;EDIT: I should add it was crippling every node, master and slave, within moments of a restart at times but not predictably. On 3.0 a step down and restart would cure a stressed node&lt;/p&gt;</comment>
                            <comment id="1217085" author="ramon.fernandez" created="Mon, 28 Mar 2016 19:15:44 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=paul.ridgway&quot; class=&quot;user-hover&quot; rel=&quot;paul.ridgway&quot;&gt;paul.ridgway&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-22831&quot; title=&quot;Low query rate with heavy cache pressure and an idle collection&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-22831&quot;&gt;&lt;del&gt;SERVER-22831&lt;/del&gt;&lt;/a&gt; will be part of 3.2.5. A 3.2.5-rc0 release candidate suitable for testing is currently scheduled for next week.&lt;/p&gt;

&lt;p&gt;I am going to mark this ticket as a duplicate of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-22831&quot; title=&quot;Low query rate with heavy cache pressure and an idle collection&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-22831&quot;&gt;&lt;del&gt;SERVER-22831&lt;/del&gt;&lt;/a&gt;. If your testing with 3.2.5-rc0 does not address this issue please post back here so we can reopen and investigate further.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="1211322" author="paul.ridgway" created="Tue, 22 Mar 2016 16:03:35 +0000"  >&lt;p&gt;Thanks - I guess as a restart helps then there are pages suitable for eviction it just cant find them fast enough?&lt;/p&gt;

&lt;p&gt;We are trialling 3.2 in dev currently so upgrade is certainly an option - do you know when it is likely to get backported?&lt;/p&gt;

&lt;p&gt;Cheers&lt;/p&gt;</comment>
                            <comment id="1211135" author="bruce.lucas@10gen.com" created="Tue, 22 Mar 2016 14:17:37 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=paul.ridgway&quot; class=&quot;user-hover&quot; rel=&quot;paul.ridgway&quot;&gt;paul.ridgway&lt;/a&gt;. This looks similar to the previous incident:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/114453/114453_cache2.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;starting about A eviction rates aren&apos;t able to keep up so cache usage grows above target levels.&lt;/li&gt;
	&lt;li&gt;&quot;pages walked for eviction&quot; is very high relative to pages evicted, and appears to have hit a ceiling at about A, so identifying pages to be evicted is the likely bottleneck. This is single-threaded, which explains why adding CPUs didn&apos;t address the problem.&lt;/li&gt;
	&lt;li&gt;as a result cache fills B-C and D-E, application threads get involved in eviction, and performance suffers: replication struggles (&quot;repl apply batches totalMillis (/s)&quot;, and operations queue.&lt;/li&gt;
	&lt;li&gt;I couldn&apos;t find a specific trigger operation in the metrics, but I imagine this is a result of some (possibly small) change in cache pressure. It appears, based on eviction statistics, to possibly have been building for some time prior to A, at which point it hit the ceiling on pages walked for eviction.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;We suspect this could be related to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-22831&quot; title=&quot;Low query rate with heavy cache pressure and an idle collection&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-22831&quot;&gt;&lt;del&gt;SERVER-22831&lt;/del&gt;&lt;/a&gt;, which is fixed in master and planned to be backported to 3.2. Would you be in a position to try 3.2 to see if it addresses this issue once that is available?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Bruce&lt;/p&gt;</comment>
                            <comment id="1210874" author="paul.ridgway" created="Tue, 22 Mar 2016 08:01:55 +0000"  >&lt;p&gt;Hi,&lt;/p&gt;

&lt;p&gt;It happened again on the same node I had stepped down at ~6AM (as a slave now) at ~07:50. 100 cache use, &apos;slow&apos; queries as a result, 59.8% system RAM usage.&lt;/p&gt;

&lt;p&gt;Logs submitted (including ss.log) as node3-22mar2016.zip.&lt;/p&gt;

&lt;p&gt;Appeared to recover following a restart.&lt;/p&gt;

&lt;p&gt;EDIT:&lt;/p&gt;

&lt;p&gt;We now log some mongo metrics with more granularity in our monitoring system, so attaching plots of cache and RAM usage - note node3 was primary, stepped down and node1 took over at 8am. Interestingly node2 is quite choppy on cache and is the only one using over 50% system RAM.&lt;/p&gt;

&lt;p&gt;Attached WT Cache usage during the incident, also attached plot for 3 nodes, colour matched WT Cache (solid) to RAM (dotted). As a reminder these VMs have 30GB of RAM.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;/p&gt;

&lt;p&gt;Paul&lt;/p&gt;</comment>
                            <comment id="1210855" author="paul.ridgway" created="Tue, 22 Mar 2016 06:16:49 +0000"  >&lt;p&gt;Hi,&lt;/p&gt;

&lt;p&gt;It happened again this morning on the Primary, possibly as early as 2AM am and stopped at 6:08AM once stepped down.&lt;/p&gt;

&lt;p&gt;Mongo system memory usage was up to 58.3% and cache use was solidly at 100% causing slow queries (which are not normally considered slow).&lt;/p&gt;

&lt;p&gt;It seems the ss.log writer crashed at some point before hand but I have iostat &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/sad.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;

&lt;p&gt;Logs submitted.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;/p&gt;

&lt;p&gt;Paul&lt;/p&gt;</comment>
                            <comment id="1208819" author="paul.ridgway" created="Sat, 19 Mar 2016 15:36:29 +0000"  >&lt;p&gt;Hi Bruce,&lt;/p&gt;

&lt;p&gt;I tried both before and after - tried a large range of min and max threads too - it all made zero impact.&lt;/p&gt;

&lt;p&gt;Cheers,&lt;/p&gt;

&lt;p&gt;Paul&lt;/p&gt;</comment>
                            <comment id="1208793" author="bruce.lucas@10gen.com" created="Sat, 19 Mar 2016 12:03:56 +0000"  >&lt;p&gt;Hi Paul,&lt;/p&gt;

&lt;p&gt;The tool used to generate the graphs is an internal tool that we don&apos;t yet feel is in a condition to make public.&lt;/p&gt;

&lt;p&gt;Thanks for the confirmation. One other question occurs to me: you mentioned that adding more eviction threads didn&apos;t help, but you also mentioned that adding CPU did help somewhat. Did you try increasing the eviction threads before or after you increased CPU? If it was before, would you be able to try more eviction threads now that you have more CPU?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Bruce&lt;/p&gt;
</comment>
                            <comment id="1208787" author="paul.ridgway" created="Sat, 19 Mar 2016 11:55:13 +0000"  >&lt;p&gt;Hi Bruce,&lt;/p&gt;

&lt;p&gt;Thanks for following up!&lt;/p&gt;

&lt;p&gt;This incident was not exactly as we&apos;ve seen before - as you say, there are some larger queries (aggregate) running.&lt;/p&gt;

&lt;p&gt;On previous incidents we&apos;ve stopped all but the core consumer services and seen no recovery.&lt;/p&gt;

&lt;p&gt;Additionally, despite letting the aggregate queries keep running a restart was enough to recover the instance and have the cache site around 80%.&lt;/p&gt;

&lt;p&gt;The memory did peak over 50% but not as bad as before.&lt;/p&gt;

&lt;p&gt;As you note, the main issue still seems to be an eviction rate that cannot keep up despite plenty of resource being available (CPU, RAM, iops, disk throughput, etc).&lt;/p&gt;

&lt;p&gt;Cheers,&lt;/p&gt;

&lt;p&gt;Paul&lt;/p&gt;

&lt;p&gt;P.S - is there a way for us to produce those plots or is it a private tool?&lt;/p&gt;</comment>
                            <comment id="1208214" author="bruce.lucas@10gen.com" created="Fri, 18 Mar 2016 18:02:23 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=paul.ridgway&quot; class=&quot;user-hover&quot; rel=&quot;paul.ridgway&quot;&gt;paul.ridgway&lt;/a&gt;, thanks for uploading that snapshot. here&apos;s a plot of some of the data from the end of the snapshot:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/114490/114490_cache1.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;We can see starting at C, at about 2016-03-17 09:13:38Z, the cache usage begins to rise, and around D at about 2016-03-17 09:16:58.828Z it reaches 100%. At this point we see application threads begin to do evictions because of the 100% cache full condition, and this would have impacted performance. Can you confirm that this corresponds to events as you observed them?&lt;/p&gt;

&lt;p&gt;Regarding your opening comment that mongod RAM usage sometimes goes as high as 80%, it doesn&apos;t appear that we are seeing that in this instance as RAM usage appears about as expected, with virtual memory around 19 GB; can you confirm?&lt;/p&gt;

&lt;p&gt;The trigger in this case appears, based on coincidence in timing, to be a number of aggregation queries that started at C, around 2016-03-17 09:13:38Z. I imagine these queries were probably accessing data not in cache so were creating increased cache pressure; does this seem plausible to you?&lt;/p&gt;

&lt;p&gt;Here are some statistics about cache churn just before and just after C:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                                  before C      after C      change&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;unmodified pages evicted (/s)     5,584         5,423        -161&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;modified pages evicted (/s)       2,928         2,774        -154&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;pages read into cache (/s)        8,519         8,781        +262&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;We see an increase in pages read into the cache, presumably a result of the aggregations; but we see a &lt;em&gt;decrease&lt;/em&gt;, both in modified and unmodified pages evicted per second, for a net increase of about 500-600 pages per second in the cache, consistent with the cache growth we see during this period. What is not clear so far however is why evictions weren&apos;t to keep up to keep up in order to keep cache usage at target levels below 100%; we will continue to look into that.&lt;/p&gt;

&lt;p&gt;Bruce&lt;/p&gt;</comment>
                            <comment id="1206426" author="paul.ridgway" created="Thu, 17 Mar 2016 09:24:30 +0000"  >&lt;p&gt;One slave did it this morning without a big query - I snapshotted the logs while it was happening and uploaded those (logs-17mar2016.zip).&lt;/p&gt;</comment>
                            <comment id="1203456" author="paul.ridgway" created="Tue, 15 Mar 2016 08:04:54 +0000"  >&lt;p&gt;I was running an unindexed query this morning which first time completed without drama in ~1000 seconds, second run (Same query) ran for an hour before I killed it, got the cache to 100% and mongo using 60% system RAM. After killing the query mongo eventually recovered. I&apos;ll submit the logs anyway though I&apos;m not certain this is the same issue - even though the symptoms are identical we were not running large unindexed queries when it last happened...&lt;/p&gt;

&lt;p&gt;I stopped the query no more than a minute or so before taking a copy of the ss.log and iostat.log files.&lt;/p&gt;</comment>
                            <comment id="1196456" author="paul.ridgway" created="Tue, 8 Mar 2016 16:21:24 +0000"  >&lt;p&gt;Thanks - I did wonder if 3.2 was better but looking at the release notes it was hard to know if WT was better or whether you back-ported critical/performance/engine fixes?&lt;/p&gt;

&lt;p&gt;Having said that, in another cluster we run 3.2 and can make it thrash the cache but it was a different workload so its not necessarily comparable.&lt;/p&gt;</comment>
                            <comment id="1196251" author="ramon.fernandez" created="Tue, 8 Mar 2016 13:50:57 +0000"  >&lt;p&gt;Understood, we&apos;ll keep this ticket open for the time being.&lt;/p&gt;

&lt;p&gt;Note that MongoDB 3.2 includes a lot of improvements over 3.0, specially on WiredTiger, so if you have the ability to test the same scenario in a 3.2 replica set it would be useful to see if this particular issue has been addressed, and whether 3.2 will be a better fit for your use case.&lt;/p&gt;

&lt;p&gt;Anyway, please let us know when you have data and have uploaded it, and we&apos;ll take a closer look.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="1196248" author="paul.ridgway" created="Tue, 8 Mar 2016 13:44:52 +0000"  >&lt;p&gt;HI &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ramon.fernandez&quot; class=&quot;user-hover&quot; rel=&quot;ramon.fernandez&quot;&gt;ramon.fernandez&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for getting back to me so quickly. I&apos;ll set the logging running and revert back to you once the issue has presented again - it could take a week or so it seems.&lt;/p&gt;

&lt;p&gt;Cheers,&lt;/p&gt;

&lt;p&gt;Paul&lt;/p&gt;</comment>
                            <comment id="1196216" author="ramon.fernandez" created="Tue, 8 Mar 2016 13:18:01 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=paul.ridgway&quot; class=&quot;user-hover&quot; rel=&quot;paul.ridgway&quot;&gt;paul.ridgway&lt;/a&gt;, in order to troubleshoot this issue we&apos;ll need some diagnostic information. I&apos;d like to ask you to run the following commands from the command line on each of your nodes:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;mongo --eval &quot;while(true) {print(JSON.stringify(db.serverStatus({tcmalloc:1}))); sleep(10000)}&quot; &amp;gt;ss.log &amp;amp;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;iostat -k -t -x 10 &amp;gt;iostat.log &amp;amp;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;These commands will create two files on each node, &lt;tt&gt;ss.log&lt;/tt&gt; and &lt;tt&gt;iostat.log&lt;/tt&gt;. Please start data collection now; when the issue appears again, let the data collection continue for a bit (as much as you can afford without impacting your production) and then &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/2dadfe92-a267-4c5c-9cfa-518b5640dc3c.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;upload the resulting files here&lt;/a&gt; (these files won&apos;t contain any sensitive information, and they&apos;ll only be available to MongoDB staff).&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="267706">SERVER-22831</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="293974">SERVER-24580</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="114460" name="cache.png" size="105766" author="paul.ridgway" created="Tue, 22 Mar 2016 15:11:51 +0000"/>
                            <attachment id="114490" name="cache1.png" size="355922" author="bruce.lucas@mongodb.com" created="Tue, 22 Mar 2016 15:58:52 +0000"/>
                            <attachment id="114453" name="cache2.png" size="339882" author="bruce.lucas@mongodb.com" created="Tue, 22 Mar 2016 14:12:56 +0000"/>
                            <attachment id="114461" name="ram-and-cache.png" size="169876" author="paul.ridgway" created="Tue, 22 Mar 2016 15:20:39 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>19.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 8 Mar 2016 13:08:51 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        7 years, 34 weeks, 4 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            7 years, 34 weeks, 4 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10020"><![CDATA[Linux]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>paul.ridgway</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrkecf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hsj5sn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hs9ynz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>