<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:44:15 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-17386] Cursor cache causes excessive memory utilization in WiredTiger</title>
                <link>https://jira.mongodb.org/browse/SERVER-17386</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;div class=&quot;panel&quot; style=&quot;background-color: #EEEEEE;border-color: #ccc;border-width: 1px;&quot;&gt;&lt;div class=&quot;panelHeader&quot; style=&quot;border-bottom-width: 1px;border-bottom-color: #ccc;background-color: #6CB33F;&quot;&gt;&lt;b&gt;Issue Status as of Jul 14, 2015&lt;/b&gt;&lt;/div&gt;&lt;div class=&quot;panelContent&quot; style=&quot;background-color: #EEEEEE;&quot;&gt;
&lt;p&gt;&lt;b&gt;ISSUE SUMMARY&lt;/b&gt;&lt;br/&gt;
WiredTiger allows MongoDB to open an unbounded number of WiredTiger sessions, which means that despite a limited number of WiredTiger cursors per session, the total number of cached sessions an cursors may lead to excessive memory use.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;USER IMPACT&lt;/b&gt;&lt;br/&gt;
MongoDB with WiredTiger may consume too much memory in some situations, leading to excessive memory use beyond the normal WiredTiger database cache, potentially triggering out of memory conditions and even the OOM killer.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;WORKAROUNDS&lt;/b&gt;&lt;br/&gt;
None.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;AFFECTED VERSIONS&lt;/b&gt;&lt;br/&gt;
MongoDB 3.0.0 through 3.0.4.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;FIX VERSION&lt;/b&gt;&lt;br/&gt;
The fix is included in the 3.0.5 production release.&lt;/p&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;h6&gt;&lt;a name=&quot;Originaldescription&quot;&gt;&lt;/a&gt;Original description&lt;/h6&gt;

&lt;p&gt;Running RC10 with the c# driver, 50 threads concurrently bulk inserting/updating/reading, I see stacktraces in the console output followed by a big speed drop. The speed drop appears to be related to reads, but I can&apos;t be sure yet.&lt;/p&gt;

&lt;p&gt;I will attempt to reproduce this with a long running test and report back.&lt;/p&gt;</description>
                <environment></environment>
        <key id="186288">SERVER-17386</key>
            <summary>Cursor cache causes excessive memory utilization in WiredTiger</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="michael.cahill@mongodb.com">Michael Cahill</assignee>
                                    <reporter username="nickj">Nick Judson</reporter>
                        <labels>
                            <label>WTmem</label>
                    </labels>
                <created>Thu, 26 Feb 2015 00:52:56 +0000</created>
                <updated>Sat, 19 Sep 2015 00:06:05 +0000</updated>
                            <resolved>Mon, 29 Jun 2015 01:53:48 +0000</resolved>
                                    <version>3.0.0-rc10</version>
                                    <fixVersion>3.0.5</fixVersion>
                    <fixVersion>3.1.5</fixVersion>
                                    <component>WiredTiger</component>
                                        <votes>3</votes>
                                    <watches>35</watches>
                                                                                                                <comments>
                            <comment id="952751" author="xgen-internal-githook" created="Mon, 29 Jun 2015 01:52:33 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;michaelcahill&apos;, u&apos;name&apos;: u&apos;Michael Cahill&apos;, u&apos;email&apos;: u&apos;michael.cahill@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-17386&quot; title=&quot;Cursor cache causes excessive memory utilization in WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-17386&quot;&gt;&lt;del&gt;SERVER-17386&lt;/del&gt;&lt;/a&gt; WiredTiger session cache improvements&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/91c9eaeae7dbe3ff5e5e5b369871562beda908a8&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/91c9eaeae7dbe3ff5e5e5b369871562beda908a8&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="952748" author="xgen-internal-githook" created="Mon, 29 Jun 2015 01:50:18 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;michaelcahill&apos;, u&apos;name&apos;: u&apos;Michael Cahill&apos;, u&apos;email&apos;: u&apos;michael.cahill@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-17386&quot; title=&quot;Cursor cache causes excessive memory utilization in WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-17386&quot;&gt;&lt;del&gt;SERVER-17386&lt;/del&gt;&lt;/a&gt; WiredTiger session cache improvements&lt;br/&gt;
Branch: v3.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/49f1cf81a2aa87792e000f73816015f3fd16b4a6&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/49f1cf81a2aa87792e000f73816015f3fd16b4a6&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="942836" author="martin.bligh" created="Wed, 17 Jun 2015 12:48:23 +0000"  >&lt;p&gt;Wondering if it&apos;s worth benchmarking together with the other intended change (unless you think it&apos;s not a good idea):&lt;br/&gt;
&quot;Limit the number of slots on WiredTigerSessionCache to less than 64 slots (4 or 8).&quot; &lt;br/&gt;
if you&apos;re not doing this already? Else we will have to run them all again in a few days &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/wink.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt; &lt;br/&gt;
Seems like a tradeoff between cache affinity and contention ... I&apos;d think we want it as small as possible without causing a lot of contention (on most loads)&lt;/p&gt;</comment>
                            <comment id="942675" author="michael.cahill" created="Wed, 17 Jun 2015 08:11:33 +0000"  >&lt;p&gt;There could be a bunch of things involved in the 128 thread case, not least of which is that we&apos;re running the client on the same machine as the server and they&apos;re competing for the same CPU.&lt;/p&gt;

&lt;p&gt;We&apos;ve been doing more runs today, including the mms.js script that &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=david.hows&quot; class=&quot;user-hover&quot; rel=&quot;david.hows&quot;&gt;david.hows&lt;/a&gt; and the hammer &lt;tt&gt;single_query.sh&lt;/tt&gt; workload.  They are showing a consistent drop in the number of open sessions (from 148 to 28 with hammer).&lt;/p&gt;

&lt;p&gt;There is still some tweaking of the cursor cache required: if I set it too small (e.g., 10 cursors), then there is approx 10% slowdown in the hammer workload as cursors are closed and reopened.  If I make the cursor cache big enough to hold all 48 cursors that operations use, then there is no performance impact of the change (in fact it seems marginally faster).  Even with that larger cache, there is still a reduction in the total number of open cursors from 7412 to 1095.&lt;/p&gt;

&lt;p&gt;I&apos;ll get this pushed into code review tomorrow: &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=david.hows&quot; class=&quot;user-hover&quot; rel=&quot;david.hows&quot;&gt;david.hows&lt;/a&gt; is still chasing a bug that I can&apos;t reproduce.&lt;/p&gt;</comment>
                            <comment id="942509" author="david.hows" created="Wed, 17 Jun 2015 00:22:21 +0000"  >&lt;p&gt;Should also add that not all is bad, there is a massive drop in resource consumption in the 128 thread case mentioned above. The session count is about accurate with what we would expect for 128 threads, the 10 leftover are those outside the cache created by various internal components on boot; 10x less cursors and 2-3x less sessions&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Vanilla MongoDB&lt;/b&gt;&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;	&quot;session&quot; : {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;		&quot;open cursor count&quot; : 12398,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;		&quot;open session count&quot; : 366&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;	},&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;&lt;b&gt;Patched&lt;/b&gt;&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;	&quot;session&quot; : {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;		&quot;open cursor count&quot; : 1398,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;		&quot;open session count&quot; : 138&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;	},&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="941641" author="martin.bligh" created="Tue, 16 Jun 2015 13:19:45 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=michael.cahill&quot; class=&quot;user-hover&quot; rel=&quot;michael.cahill&quot;&gt;michael.cahill&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=david.hows&quot; class=&quot;user-hover&quot; rel=&quot;david.hows&quot;&gt;david.hows&lt;/a&gt; - sounds good. Per discussion yesterday (including Alex) I think we expect some minor perf regressions in exchange for stability, so shouldn&apos;t be a blocking problem, but probably worth poking at them a bit to check causes etc. (eg in the 1 thread case above?)&lt;/p&gt;

&lt;p&gt;Per David&apos;s last comment, is the 128-thread case the sched_yield thing? ISTR --nojournal also making a large difference at high thread counts for hammer insert&lt;/p&gt;</comment>
                            <comment id="941465" author="david.hows" created="Tue, 16 Jun 2015 07:42:30 +0000"  >&lt;p&gt;I&apos;ve got some initial changes put together for #2 and #4 of Martin&apos;s list, but still need to do a little more work to confirm if there is a performance impact and if so, look at how to avoid a performance regression.&lt;/p&gt;

&lt;p&gt;The current benchmarks i&apos;ve done show the following (num threads vs execution time) so I want to investigate the high thread number cases as those seem to be where things look like a regression.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Patched&lt;/b&gt;&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;1	14362.742176929787&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;4	44660.264345779105&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;16	45461.05404115101&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;128	29478.12966445289&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;&lt;b&gt;Vanilla&lt;/b&gt;&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;1	16118.435705814753&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;4	46082.747979357846&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;16	45675.32389163103&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;128	28135.928187454814&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="941341" author="michael.cahill" created="Tue, 16 Jun 2015 01:59:13 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=martin.bligh&quot; class=&quot;user-hover&quot; rel=&quot;martin.bligh&quot;&gt;martin.bligh&lt;/a&gt;, I&apos;m planning to work with &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=david.hows&quot; class=&quot;user-hover&quot; rel=&quot;david.hows&quot;&gt;david.hows&lt;/a&gt; on 2 and 4.  He&apos;s got preliminary code with a non-locking implementation of a single-slot session cache.&lt;/p&gt;

&lt;p&gt;We&apos;ll report back with the impact of these changes on memory use and performance for some set of workloads.  Will that get in your way?&lt;/p&gt;</comment>
                            <comment id="940886" author="martin.bligh" created="Mon, 15 Jun 2015 19:20:20 +0000"  >&lt;p&gt;Intent for 3.0 after discussion between WT and Storage teams.&lt;/p&gt;

&lt;p&gt;1. Limit the number of slots on WiredTigerSessionCache to less than 64 slots (4 or 8). May be a slight perf impact, but order of magnitude less memory consumption.&lt;br/&gt;
2. Bound number of WT cursors bound per session to 16 &lt;b&gt;on release&lt;/b&gt;.  (zero may also be a possibility)&lt;br/&gt;
3. Idle client cursors should not hold storage engine (or WT) resources.&lt;br/&gt;
4. Investigate picking more efficiently which slot to use in WiredTigerSessionCache (David has a patch which seems to have no downside ...)&lt;/p&gt;

&lt;p&gt;All numbers are subject to change post-benchmarking &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/wink.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;

&lt;p&gt;Other thoughts for further investigation include:&lt;/p&gt;

&lt;p&gt;A. Remove the WiredTigerSession cache by tying the WiredTigerSession directly to the Mongo Session&lt;br/&gt;
B. Making the WiredTigerSessionCache a single &quot;slot&quot; again and changing the data structure to something more performant instead.&lt;/p&gt;</comment>
                            <comment id="859827" author="dan@10gen.com" created="Fri, 20 Mar 2015 22:00:28 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mdcallag&quot; class=&quot;user-hover&quot; rel=&quot;mdcallag&quot;&gt;mdcallag&lt;/a&gt;, only linux uses tcmalloc in 3.0.  Previously osx also used tcmalloc.&lt;/p&gt;</comment>
                            <comment id="853114" author="mdcallag" created="Mon, 16 Mar 2015 17:57:40 +0000"  >&lt;p&gt;Just curious, does this mean mongod doesn&apos;t using tcmalloc or jemalloc for Windows?&lt;br/&gt;
&quot;Windows has a single lock on its heap. I even ran the repro with two heaps in mongod, one heap for mongodb, and a separate heap for WT.&quot;&lt;/p&gt;</comment>
                            <comment id="839742" author="mark.benvenuto" created="Sun, 1 Mar 2015 02:11:22 +0000"  >&lt;p&gt;The repro consists of about 100 connections with a mix of inserts, and updates across 150 collections in a single database.&lt;/p&gt;

&lt;p&gt;There are a few issues. The &quot;memory leak&quot; occurs because of a session &amp;amp; cursor &quot;leak&quot;. If we fix the session/cursor leak, there may be other issues though.&lt;/p&gt;

&lt;p&gt;We can see the leak from the WT counters. These are the current open sessions, and open cursors counters. These are not the sessions/cursors currently in use in a transaction (that count would be lower), just the ones that are open. Note, these counters are accurate, there appear to be no bugs in the counters themselves. I instrumented the code to confirm the session count for instance.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;       &quot;session&quot; : {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;               &quot;open cursor count&quot; : 125461,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;               &quot;open session count&quot; : 446&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;       },&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;&lt;b&gt;Results of Investigations&lt;/b&gt;&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Heavy Windows Heap Contention after the cache size is reached&lt;/li&gt;
	&lt;li&gt;WiredTigerSessionCache has significant performance problems, and suffers from cache anti-patterns.&lt;/li&gt;
	&lt;li&gt;WiredTigerSession cursors needs to be redesigned.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;&lt;b&gt;Heap Contention&lt;/b&gt;&lt;br/&gt;
Windows has a single lock on its heap. I even ran the repro with two heaps in mongod, one heap for mongodb, and a separate heap for WT. The primary source of all the heap contention is WT, not the rest of mongod&apos;s allocations. We have seen this before in some of other testing we have done, it is an opportunity for improvement, but not a blocking issue at the moment.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;WiredTigerSessionCache&lt;/b&gt;&lt;br/&gt;
The WiredTigerSessionCache is trying to cache WT_SESSION objects from WT. The problem is that as part of the work to avoid lock contention, its effectiveness as a cache has suffered.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        const int cachePartition = cachePartitionGen.addAndFetch(1) % NumSessionCachePartitions;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;The cache is divided into 64 chunks. This one line ensures that a thread will never hit the same session cache entry twice. It also means that if a customer had a single threaded application, we would go through 64 WT_SESSIONS before a thread ever actually reuses a single session since there is no cpu or thread locality into the cache. The &lt;tt&gt;cachePartitionGen.addAndFetch(1)&lt;/tt&gt; might as well be a call to random.&lt;/p&gt;

&lt;p&gt;Also, in this repro, the number of open sessions has grown to 446 yet there are only 100 connections. This also shows how the round robin between buckets has caused the session count to greatly exceed the number of connections.&lt;/p&gt;

&lt;p&gt;A better approach would be &lt;tt&gt;GetCurrentProcessorNumber&lt;/tt&gt;/&lt;tt&gt;sched_cpu&lt;/tt&gt;. See this &lt;a href=&quot;http://www.1024cores.net/home/lock-free-algorithms/tricks/per-processor-data&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;page&lt;/a&gt; for other alternatives.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;WiredTigerSession Cursor Cache&lt;/b&gt;&lt;br/&gt;
The cursor cache is designed as a &lt;tt&gt;map&amp;lt;wt table id, vector&amp;lt; cursor &amp;gt;  &amp;gt;&lt;/tt&gt; where the vector is capped at 10 cursors. The map itself is not limited in size. This means that cursor cache can have a max size be &lt;tt&gt;10 * N&lt;/tt&gt; where N is the number of WT tables (i.e., mongo collections + indexes). This cursor cache is per WiredTigerSesson, and since there can be an unbounded number of WiredTigerSession, the total number of open cursors can grow very large.&lt;/p&gt;

&lt;p&gt;Ideally, a MRU data structure would be used instead like those available in Boost &lt;a href=&quot;http://www.boost.org/doc/libs/1_57_0/libs/multi_index/example/serialization.cpp&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Multi Index&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="838618" author="dan@10gen.com" created="Fri, 27 Feb 2015 01:17:38 +0000"  >&lt;p&gt;mongod should not generally be using that much extra memory over configured cache size.  If you are able to modify the test program from &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-17299&quot; title=&quot;Extraneous memory allocations in snappy hurt WiredTiger performance on Windows&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-17299&quot;&gt;&lt;del&gt;SERVER-17299&lt;/del&gt;&lt;/a&gt; in a way which can trigger this condition, that would extremely helpful of course.  If you can compress and attach the log file from a run which hits this issue, that would also be helpful.&lt;/p&gt;

&lt;p&gt;Thanks again for the report.&lt;/p&gt;</comment>
                            <comment id="838333" author="nickj" created="Thu, 26 Feb 2015 20:02:49 +0000"  >&lt;p&gt;And one more piece of information - the stack traces always seem to happen at shutdown, and seem to happen twice. The last of the two appears immediately before the ^C is logged (in my log files).&lt;/p&gt;</comment>
                            <comment id="838299" author="nickj" created="Thu, 26 Feb 2015 19:42:51 +0000"  >&lt;p&gt;So mongod must have some leeway with the --wiredTigerCacheSizeGB flag. As a test I set it to 2GB and it&apos;s now at 2.9 GB &lt;span class=&quot;error&quot;&gt;&amp;#91;and now up to 4GB&amp;#93;&lt;/span&gt; in both mongostat and task manager.&lt;/p&gt;

&lt;p&gt;Not sure if this is expected or perhaps a slow memory leak?&lt;/p&gt;</comment>
                            <comment id="838222" author="nickj" created="Thu, 26 Feb 2015 18:31:14 +0000"  >&lt;p&gt;I have been able to repro this. Here are some details:&lt;/p&gt;

&lt;p&gt;-wiredTiger rc10&lt;br/&gt;
-i7 5960X, 16gb, SSDs&lt;br/&gt;
-Win 8.1 64bit&lt;/p&gt;

&lt;p&gt;I&apos;m using the following command line:&lt;/p&gt;

&lt;p&gt;mongod --dbpath=e:\mongo --storageEngine wiredTiger --wiredTigerCacheSizeGB=6 --logpath-c:\mongo\mongo.log&lt;/p&gt;

&lt;p&gt;It appears that mongod is using &amp;gt; 6GB (see screenshot), and so system paging kicks in when it hits ~9GB. This, obviously, slows things down quite a bit, &lt;del&gt;but it also seems to coincide with the stacktrace in the first screenshot&lt;/del&gt;. See below, the exceptions appear to be thrown at shutdown...I just happen to stop mongod when speeds drop.&lt;/p&gt;

&lt;p&gt;I have a 110MB log file, which shows exactly the same as the existing screenshot.&lt;/p&gt;</comment>
                            <comment id="838111" author="nickj" created="Thu, 26 Feb 2015 17:17:17 +0000"  >&lt;p&gt;I don&apos;t think I have any logs - this was run in the console window without a log target argument. Would MMS have anything useful? I will attempt to repro when I get a minute (with logging to a file).&lt;/p&gt;

&lt;p&gt;I can&apos;t share the code unfortunately. I have a sample which is attached to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-17299&quot; title=&quot;Extraneous memory allocations in snappy hurt WiredTiger performance on Windows&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-17299&quot;&gt;&lt;del&gt;SERVER-17299&lt;/del&gt;&lt;/a&gt; but it isn&apos;t a full representation of what I&apos;m doing. I&apos;ll see if I can build it out for you.&lt;/p&gt;</comment>
                            <comment id="837938" author="ramon.fernandez" created="Thu, 26 Feb 2015 14:26:55 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=nickj&quot; class=&quot;user-hover&quot; rel=&quot;nickj&quot;&gt;nickj&lt;/a&gt;, can you please upload full server logs in text? Also, is it possible for you to share the code you&apos;re using to trigger these messages so we can try to reproduce them on our end?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="187365">SERVER-17456</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="208264">SERVER-18821</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="214639">SERVER-19273</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="222480">SERVER-19562</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="185927">SERVER-17364</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="210757">NODE-487</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="186885">SERVER-17424</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="65785" name="mongo.png" size="146067" author="nickj" created="Thu, 26 Feb 2015 00:52:56 +0000"/>
                            <attachment id="65868" name="mongoMem.png" size="48097" author="nickj" created="Thu, 26 Feb 2015 18:31:14 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>18.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_12451" key="com.atlassian.jira.plugin.system.customfieldtypes:multiversion">
                        <customfieldname>Backport Completed</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="15441">3.0.5</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 26 Feb 2015 14:26:55 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        8 years, 33 weeks, 3 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/WRITING-1375'>WRITING-1375</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            8 years, 33 weeks, 3 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>dan@mongodb.com</customfieldvalue>
            <customfieldvalue>david.hows</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>mark.benvenuto@mongodb.com</customfieldvalue>
            <customfieldvalue>mdcallag</customfieldvalue>
            <customfieldvalue>martin.bligh</customfieldvalue>
            <customfieldvalue>michael.cahill@mongodb.com</customfieldvalue>
            <customfieldvalue>nickj</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrika7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hren53:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>164025</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="470">Quint Iteration 3.1.2</customfieldvalue>
    <customfieldvalue id="490">Quint Iteration 3</customfieldvalue>
    <customfieldvalue id="536">Quint Iteration 5</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11861" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>User Summary</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="11858"><![CDATA[Completed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hriyxj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>