<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:00:32 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-22482] Cache growing to 100% followed by crash</title>
                <link>https://jira.mongodb.org/browse/SERVER-22482</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;The primary server on my primary shard has encountered crashing problems repeatedly during production. Mongostat reports the used% of cache growing to 100% (and sometimes 101%) and the dirty % to over 90%. When this situation occurs it is just a matter of time until the server crashes. Memory size and res do not grow to the point where the server crashes because it is out of memory.&lt;/p&gt;

&lt;p&gt;Opening the log file for either server (this happened to both the promoted secondary as well as the primary) I find thousands of lines with these error messages:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-02-04T23:46:00.871+0000 I STORAGE  [conn40] WTIndex::updatePosition -- the new key ( 2E15D5ECA4877FFFFEB27DF76BB90446AE2462) is less than the previous key (2E4B485BC8877FFFFEAD53C76DEB044B83D4DA), which is a bug.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-02-04T23:46:00.871+0000 I STORAGE  [conn40] WTIndex::updatePosition -- the new key ( 140402C0) is less than the previous key (3C0A3238324B204A4F45203A536F43616C4A4F4542000447951242), which is a bug.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-02-04T23:46:00.871+0000 I STORAGE  [conn40] WTIndex::updatePosition -- the new key ( 2E15D5F308EB0442B6EA52) is less than the previous key (2E4B485BC8877FFFFEAD526FF19B044B83B3EA), which is a bug.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-02-04T23:46:00.871+0000 I STORAGE  [conn40] WTIndex::updatePosition -- the new key ( 2E15D5F3082E2A7DC42278800001464CDE2D0678800001464CDE2D060442B6EA52) is less than the previous key (2E4B485BC82E01D860A07880000152AC3892147880000152AC389214044B83D492), which is a bug.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-02-04T23:46:00.871+0000 I STORAGE  [conn40] WTIndex::updatePosition -- the new key ( 2E15D5F308877FFFFEB9B321D2F90442B6EA52) is less than the previous key (2E4B485BC8877FFFFEAD53C76DEB044B83D4DA), which is a bug.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-02-04T23:46:00.871+0000 I STORAGE  [conn40] WTIndex::updatePosition -- the new key ( 140402E8) is less than the previous key (3C0A3238324B204A4F45203A536F43616C4A4F4542000447951242), which is a bug.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-02-04T23:46:00.871+0000 I STORAGE  [conn40] WTIndex::updatePosition -- the new key ( 2E15D5F478EB04445670AA) is less than the previous key (2E4B485BC8877FFFFEAD526FF19B044B83B3EA), which is a bug.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-02-04T23:46:00.871+0000 I STORAGE  [conn40] WTIndex::updatePosition -- the new key ( 2E15D5F4782D317A2E7880000149E9C7C8D27880000149E9C7C8D204445670AA) is less than the previous key (2E4B485BC82E01D860A07880000152AC3892147880000152AC389214044B83D492), which is a bug.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-02-04T23:46:00.871+0000 I STORAGE  [conn40] WTIndex::updatePosition -- the new key ( 2E15D5F478877FFFFEB61638372D04445670AA) is less than the previous key (2E4B485BC8877FFFFEAD53C76DEB044B83D4DA), which is a bug.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-02-04T23:46:00.871+0000 I STORAGE  [conn40] WTIndex::updatePosition -- the new key ( 14040300) is less than the previous key (3C0A3238324B204A4F45203A536F43616C4A4F4542000447951242), which is a bug.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-02-04T23:46:00.871+0000 I STORAGE  [conn40] WTIndex::updatePosition -- the new key ( 2E15D5F5FA877FFFFEAD520BCC4D0445D35EFA) is less than the previous key (2E4B485BC8877FFFFEAD526FF19B044B83B3EA), which is a bug.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-02-04T23:46:00.871+0000 I STORAGE  [conn40] WTIndex::updatePosition -- the new key ( 2E15D5F5FA2D27E306788000014BFD075515788000014BFD07551504453ACEF2) is less than the previous key (2E4B485BC82E01D860A07880000152AC3892147880000152AC389214044B83D492), which is a bug.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-02-04T23:46:00.871+0000 I STORAGE  [conn40] WTIndex::updatePosition -- the new key ( 2E15D5F5FA877FFFFEB06628460F044880E17A) is less than the previous key (2E4B485BC8877FFFFEAD53C76DEB044B83D4DA), which is a bug.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-02-04T23:46:00.871+0000 I STORAGE  [conn40] WTIndex::updatePosition -- the new key ( 14040308) is less than the previous key (3C0A3238324B204A4F45203A536F43616C4A4F4542000447951242), which is a bug.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-02-04T23:46:00.871+0000 I STORAGE  [conn40] WTIndex::updatePosition -- the new key ( 2E15D5F5FA877FFFFEAD5271F85C04475D1322) is less than the previous key (2E4B485BC8877FFFFEAD526FF19B044B83B3EA), which is a bug.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-02-04T23:46:00.871+0000 I STORAGE  [conn40] WTIndex::updatePosition -- the new key ( 2E15D5F5FA2E013EB10478800001497BF3CA7378800001497BF3CA73044431CE52) is less than the previous key (2E4B485BC82E01D860A07880000152AC3892147880000152AC389214044B83D492), which is a bug.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-02-04T23:46:00.870+0000 I STORAGE  [conn42] WTIndex::updatePosition -- the new key ( 2E15D5F5FAEB0442EEBC72) is less than the previous key (2E4B485BC8877FFFFEAD526FF19B044B83B41A), which is a bug.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-02-04T23:46:00.872+0000 I STORAGE  [conn40] WTIndex::updatePosition -- the new key ( 2E15D5F5FA877FFFFEB1AD52F4D804475D1322) is less than the previous key (2E4B485BC8877FFFFEAD53C76DEB044B83D4DA), which is a bug.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-02-04T23:46:00.872+0000 I STORAGE  [conn42] WTIndex::updatePosition -- the new key ( 2E15D5F5FA2EC2DA5340788000014E52AD0B27788000014E52AD0B2704475D1322) is less than the previous key (2E4B485BC82E01DE84C27880000152AA3D0BAB7880000152AA3D0BAB044B8272B2), which is a bug.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-02-04T23:46:00.872+0000 I STORAGE  [conn42] WTIndex::updatePosition -- the new key ( 2E15D5F5FA877FFFFEB9156C2AD90442EEBC72) is less than the previous key (2E4B485BC8877FFFFEAD53C76DEB044B83D4EA), which is a bug.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;The end of the log has no crash information at all.&lt;/p&gt;</description>
                <environment></environment>
        <key id="263278">SERVER-22482</key>
            <summary>Cache growing to 100% followed by crash</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="kelsey.schubert@mongodb.com">Kelsey Schubert</assignee>
                                    <reporter username="miketempleman">Mike Templeman</reporter>
                        <labels>
                    </labels>
                <created>Fri, 5 Feb 2016 01:14:19 +0000</created>
                <updated>Fri, 12 Feb 2016 20:34:11 +0000</updated>
                            <resolved>Fri, 12 Feb 2016 18:40:43 +0000</resolved>
                                    <version>3.2.1</version>
                                                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="1172219" author="ramon.fernandez" created="Fri, 12 Feb 2016 20:34:11 +0000"  >&lt;p&gt;Hi Mike,&lt;/p&gt;

&lt;p&gt;the 3.2.2 release is currently scheduled for next week. The 3.2.2-rc2 release candidate should have the same bits.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="1172194" author="mike@meshfire.com" created="Fri, 12 Feb 2016 20:12:27 +0000"  >&lt;p&gt;Thanks Thomas.&lt;/p&gt;

&lt;p&gt;Just as a fyi, &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-22279&quot; title=&quot;SubplanStage fails to register its MultiPlanStage&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-22279&quot;&gt;&lt;del&gt;SERVER-22279&lt;/del&gt;&lt;/a&gt; errors come in large sequences that quickly&lt;br/&gt;
fill up the cache and essentially stop the server from processing until the&lt;br/&gt;
cache is reduced, which can only happen when the errors stop. The only&lt;br/&gt;
workaround I have found so far is to step the server down and restart it.&lt;/p&gt;


&lt;p&gt;Do you have an expected date for releasing 3.2.2?&lt;/p&gt;

&lt;p&gt;&amp;#8211; &lt;/p&gt;

&lt;p&gt;&lt;b&gt;Mike Templeman&lt;/b&gt;&lt;br/&gt;
&lt;b&gt;Head of Development&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;T: @talkingfrog1950 &amp;lt;&lt;a href=&quot;http://twitter.com/missdestructo&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://twitter.com/missdestructo&lt;/a&gt;&amp;gt;&lt;br/&gt;
T: @Meshfire &amp;lt;&lt;a href=&quot;http://twitter.com/meshfire&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://twitter.com/meshfire&lt;/a&gt;&amp;gt;&lt;/p&gt;


&lt;p&gt;On Fri, Feb 12, 2016 at 10:41 AM, Thomas Schubert (JIRA) &amp;lt;jira@mongodb.org&amp;gt;&lt;/p&gt;
</comment>
                            <comment id="1172066" author="thomas.schubert" created="Fri, 12 Feb 2016 18:40:43 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=miketempleman&quot; class=&quot;user-hover&quot; rel=&quot;miketempleman&quot;&gt;miketempleman&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for opening this ticket.  I&apos;m sorry you have run into these issues. I&apos;d like to summarize our findings:&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-22279&quot; title=&quot;SubplanStage fails to register its MultiPlanStage&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-22279&quot;&gt;&lt;del&gt;SERVER-22279&lt;/del&gt;&lt;/a&gt; corrects the &lt;tt&gt;WTIndex::updatePosition&lt;/tt&gt; errors that you are experiencing&lt;/li&gt;
&lt;/ul&gt;


&lt;ul&gt;
	&lt;li&gt;Additionally, &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-21988&quot; title=&quot;Rollback does not wait for applier to finish before starting&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-21988&quot;&gt;&lt;del&gt;SERVER-21988&lt;/del&gt;&lt;/a&gt; fixes the &lt;tt&gt;Fatal assertion 28723&lt;/tt&gt; error that you experienced.&lt;/li&gt;
&lt;/ul&gt;


&lt;ul&gt;
	&lt;li&gt;Finally, &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-20306&quot; title=&quot;75% excess memory usage under WiredTiger during stress test&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-20306&quot;&gt;&lt;del&gt;SERVER-20306&lt;/del&gt;&lt;/a&gt; explains the 5 GB of free memory. Please watch &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-20306&quot; title=&quot;75% excess memory usage under WiredTiger during stress test&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-20306&quot;&gt;&lt;del&gt;SERVER-20306&lt;/del&gt;&lt;/a&gt; for updates as we work to correct this issue.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Both &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-22279&quot; title=&quot;SubplanStage fails to register its MultiPlanStage&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-22279&quot;&gt;&lt;del&gt;SERVER-22279&lt;/del&gt;&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-21988&quot; title=&quot;Rollback does not wait for applier to finish before starting&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-21988&quot;&gt;&lt;del&gt;SERVER-21988&lt;/del&gt;&lt;/a&gt; are included 3.2.2-rc2 release candidate, which is now available for &lt;a href=&quot;https://www.mongodb.org/downloads#development&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;download&lt;/a&gt; if you would like to test them. &lt;/p&gt;

&lt;p&gt;Since we have identified the causes for this behavior, I will be closing this ticket as a duplicate of these tickets. &lt;/p&gt;

&lt;p&gt;If you see WT cache growing significantly past its limit or if you experience any other issues that cannot be explained by these tickets, please do not hesitate to open a new ticket.&lt;/p&gt;

&lt;p&gt;Thank you again,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1165520" author="mike@meshfire.com" created="Fri, 5 Feb 2016 21:53:27 +0000"  >&lt;p&gt;Bruce&lt;/p&gt;

&lt;p&gt;I will set the cache size on all instances to the default. I can see that&lt;br/&gt;
the WT cache often grows beyond the set amount so allowing the default will&lt;br/&gt;
not leave GB of memory unused.&lt;/p&gt;

&lt;p&gt;I am surprised by the number of connections in comparison to the other&lt;br/&gt;
shards. I will need to dig into that to find out what is going on. Every&lt;br/&gt;
application server has it&apos;s own mongos server as well as it&apos;s own&lt;br/&gt;
connection pool. The primary shard will, of course, get more queries and&lt;br/&gt;
updates but 4X more than the other shards seems excessive.&lt;/p&gt;

&lt;p&gt;Finally and most importantly, please accept my personal apology for my&lt;br/&gt;
unprofessional message. No excuse for it.&lt;/p&gt;

&lt;p&gt;Mike&lt;/p&gt;

&lt;p&gt;&amp;#8211; &lt;/p&gt;

&lt;p&gt;&lt;b&gt;Mike Templeman&lt;/b&gt;&lt;br/&gt;
&lt;b&gt;Head of Development&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;T: @talkingfrog1950 &amp;lt;&lt;a href=&quot;http://twitter.com/missdestructo&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://twitter.com/missdestructo&lt;/a&gt;&amp;gt;&lt;br/&gt;
T: @Meshfire &amp;lt;&lt;a href=&quot;http://twitter.com/meshfire&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://twitter.com/meshfire&lt;/a&gt;&amp;gt;&lt;/p&gt;


</comment>
                            <comment id="1165516" author="mike@meshfire.com" created="Fri, 5 Feb 2016 21:47:27 +0000"  >&lt;p&gt;Ramon&lt;/p&gt;

&lt;p&gt;Can you delete my last message? It was both unprofessional and unfair. No&lt;br/&gt;
excuse for it. Please extend my apologies to anyone on the teams who might&lt;br/&gt;
have encountered it.&lt;/p&gt;

&lt;p&gt;Thanks&lt;/p&gt;

&lt;p&gt;&amp;#8211; &lt;/p&gt;

&lt;p&gt;&lt;b&gt;Mike Templeman&lt;/b&gt;&lt;br/&gt;
&lt;b&gt;Head of Development&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;T: @talkingfrog1950 &amp;lt;&lt;a href=&quot;http://twitter.com/missdestructo&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://twitter.com/missdestructo&lt;/a&gt;&amp;gt;&lt;br/&gt;
T: @Meshfire &amp;lt;&lt;a href=&quot;http://twitter.com/meshfire&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://twitter.com/meshfire&lt;/a&gt;&amp;gt;&lt;/p&gt;


&lt;p&gt;On Thu, Feb 4, 2016 at 5:27 PM, Ramon Fernandez (JIRA) &amp;lt;jira@mongodb.org&amp;gt;&lt;/p&gt;
</comment>
                            <comment id="1164916" author="bruce.lucas@10gen.com" created="Fri, 5 Feb 2016 14:42:58 +0000"  >&lt;p&gt;Hi Mike,&lt;/p&gt;

&lt;p&gt;We&apos;ve reviewed the data and it appears that&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;we see frequent crashes after virtual memory usage grows to 32 GB; this is probably the o/s OOM killer&lt;/li&gt;
	&lt;li&gt;the memory is accounted for by
	&lt;ul&gt;
		&lt;li&gt;about 25 GB in WT cache&lt;/li&gt;
		&lt;li&gt;around 5 GB of free memory; this is like memory fragmentation, tracked by &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-20306&quot; title=&quot;75% excess memory usage under WiredTiger during stress test&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-20306&quot;&gt;&lt;del&gt;SERVER-20306&lt;/del&gt;&lt;/a&gt;&lt;/li&gt;
		&lt;li&gt;an additional ~2GB of virtual memory, probably related to the nearly 2k connections&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Given this, it doesn&apos;t appear there are any additional issues beyond what we are aware of related to memory usage, and reducing the WT cache to its default value should eliminate the OOM crashes. (Note by the way that reducing WT cache does not necessarily reduce performance, and in fact under some conditions can improve it, because the remaining memory is used by the kernel as a filesystem cache, and that can have an even greater benefit than WT cache.)&lt;/p&gt;</comment>
                            <comment id="1164875" author="ramon.fernandez" created="Fri, 5 Feb 2016 14:05:19 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=miketempleman&quot; class=&quot;user-hover&quot; rel=&quot;miketempleman&quot;&gt;miketempleman&lt;/a&gt;, my colleagues in the Replication team tell me that the &lt;tt&gt;Fatal assertion 28723&lt;/tt&gt; error you&apos;re seeing in the logs has been fixed in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-21988&quot; title=&quot;Rollback does not wait for applier to finish before starting&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-21988&quot;&gt;&lt;del&gt;SERVER-21988&lt;/del&gt;&lt;/a&gt;, and the fix is included in 3.2.2, so if you upgrade the affected node(s) to 3.2.2-rc0 the problem should go away.&lt;/p&gt;

&lt;p&gt;I looked at the diagnostic data and it doesn&apos;t seem like there were any other problems going on, but I&apos;d still recommend you consider using the default cache size for the WiredTiger engine.&lt;/p&gt;

&lt;p&gt;I&apos;d understand if your decision to move back to 3.0 was firm, but I&apos;d still like to encourage you to give 3.2.2-rc0 a try.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="1164859" author="ramon.fernandez" created="Fri, 5 Feb 2016 13:45:55 +0000"  >&lt;p&gt;Thanks for the additional information &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=miketempleman&quot; class=&quot;user-hover&quot; rel=&quot;miketempleman&quot;&gt;miketempleman&lt;/a&gt;. I&apos;m sorry to hear that the experience with 3.2.1 hasn&apos;t been good so far, and I hope we can get to the bottom if these issues to make MongoDB 3.2 better. If you want to downgrade I&apos;d recommend 3.0.9, the latest stable release in the 3.0 series.&lt;/p&gt;

&lt;p&gt;The first thing I need to mention is that &lt;a href=&quot;https://docs.mongodb.org/manual/faq/storage/#to-what-size-should-i-set-the-wiredtiger-cache&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;it&apos;s not recommended to increase the WiredTiger cache size&lt;/a&gt;: this is the size limit &lt;em&gt;for the WiredTiger cache only&lt;/em&gt; &amp;#8211; &lt;tt&gt;mongod&lt;/tt&gt; will use additional memory, and increasing the cache size may reduce performance and increase the chances that the OOM killer will terminate &lt;tt&gt;mongod&lt;/tt&gt;.&lt;/p&gt;

&lt;p&gt;That being said, the log does show a fatal assertion that needs a closer look:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-02-05T03:20:44.691+0000 I REPL     [ReplicationExecutor] syncing from: s2r2.db.meshfire.com:27017&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-02-05T03:20:44.693+0000 I REPL     [SyncSourceFeedback] setting syncSourceFeedback to s2r2.db.meshfire.com:27017&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-02-05T03:20:44.696+0000 I ASIO     [NetworkInterfaceASIO-BGSync-0] Successfully connected to s2r2.db.meshfire.com:27017&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-02-05T03:20:44.697+0000 I REPL     [rsBackgroundSync] starting rollback: OplogStartMissing our last op time fetched: (term: 12, timestamp: Feb  5 01:56:40:9). source&apos;s GTE: (term: 13, timestamp: Feb  5 01:56:43:1) hashes: (-6186595901470975452/13223962844377986)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-02-05T03:20:44.698+0000 F REPL     [rsBackgroundSync] need to rollback, but in inconsistent state&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-02-05T03:20:44.698+0000 I -        [rsBackgroundSync] Fatal assertion 28723 UnrecoverableRollbackError need to rollback, but in inconsistent state. minvalid: (term: 13, timestamp: Feb  5 01:56:45:86) &amp;gt; our last optime: (term: 12, timestamp: Feb  5 01:56:40:9) @ 18750&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-02-05T03:20:44.698+0000 I -        [rsBackgroundSync] &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;***aborting after fassert() failure&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;I&apos;m also looking at the diagnostics data, please continue to watch this ticket for updates.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="1164578" author="miketempleman" created="Fri, 5 Feb 2016 03:22:43 +0000"  >&lt;p&gt;Log of failed restart on secondary due to oplog inconsistencies&lt;/p&gt;</comment>
                            <comment id="1164539" author="miketempleman" created="Fri, 5 Feb 2016 01:41:37 +0000"  >&lt;p&gt;files in &amp;lt;data&amp;gt;/diagnostic.data/&lt;/p&gt;</comment>
                            <comment id="1164538" author="miketempleman" created="Fri, 5 Feb 2016 01:37:29 +0000"  >&lt;p&gt;Configuration is an AWS ec2 i2.xlarge instance (4 virtual cpu&apos;s, 30.5GB RAM, 800GB SSD ephemeral) running AWS Linux (ami-1ecae776). The data is stored on the ephemeral drive.&lt;/p&gt;

&lt;p&gt;Startup options are:- -storageEngine wiredTiger --wiredTigerCacheSizeGB 24 --logpath xxxxx --oplogSize 90000 --fork --replSet xxxxx&lt;/p&gt;

&lt;p&gt;The log file of the server after restart is for a secondary. Are you sure you want that log?&lt;/p&gt;</comment>
                            <comment id="1164534" author="ramon.fernandez" created="Fri, 5 Feb 2016 01:26:33 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=miketempleman&quot; class=&quot;user-hover&quot; rel=&quot;miketempleman&quot;&gt;miketempleman&lt;/a&gt;, I&apos;m not sure the behavior you describe (cache growth and crash) is related to the messages you&apos;re seeing in the logs.&lt;/p&gt;

&lt;p&gt;The &lt;tt&gt;WTIndex::updatePosition&lt;/tt&gt; errors you&apos;re seeing look like a manifestation of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-22279&quot; title=&quot;SubplanStage fails to register its MultiPlanStage&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-22279&quot;&gt;&lt;del&gt;SERVER-22279&lt;/del&gt;&lt;/a&gt;, which was fixed recently in 3.2.2. A 3.2.2-rc0 release candidate was released today and it&apos;s &lt;a href=&quot;https://www.mongodb.org/downloads#development&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;available for download&lt;/a&gt;, so I&apos;d like to ask you to test it to address that issue.&lt;/p&gt;

&lt;p&gt;As for the cache growth and crash, please send us the following information so we can investigate further:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;full logs for the affected node(s) since the last restart&lt;/li&gt;
	&lt;li&gt;the contents of the &lt;tt&gt;&amp;lt;dbpath&amp;gt;/diagnostics.data&lt;/tt&gt; directory&lt;/li&gt;
	&lt;li&gt;the configuration details for the machine hosting this &lt;tt&gt;mongod&lt;/tt&gt; (CPU, available RAM, etc.)&lt;/li&gt;
	&lt;li&gt;the contents of your &lt;tt&gt;mongod.conf&lt;/tt&gt; file, or a list of configuration options used to run this &lt;tt&gt;mongod&lt;/tt&gt;&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;If the end of the log has no information at all it&apos;s possible that the &lt;tt&gt;mongod&lt;/tt&gt; process was killed by the OS, which can happen if a but allows the cache to grow past its configured limit. The information requested above should point us in the right direction.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="228592">SERVER-20306</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="244831">SERVER-21988</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="260240">SERVER-22279</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="107871" name="diagnostics.zip" size="91800707" author="miketempleman" created="Fri, 5 Feb 2016 01:41:37 +0000"/>
                            <attachment id="107940" name="ftdc.png" size="198533" author="ramon.fernandez@mongodb.com" created="Fri, 5 Feb 2016 15:13:46 +0000"/>
                            <attachment id="107874" name="shard2-crash.log" size="17002" author="miketempleman" created="Fri, 5 Feb 2016 03:22:43 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>12.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 5 Feb 2016 01:26:33 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        8 years, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            8 years, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
            <customfieldvalue>mike@meshfire.com</customfieldvalue>
            <customfieldvalue>miketempleman</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrkien:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hsi31j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;Start primary shard server during a busy day and wait a couple of hours&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[kelsey.schubert@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsf9fb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>