<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:53:10 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-20104] WT high memory usage due to high amount of free memory accumulated by TCMalloc</title>
                <link>https://jira.mongodb.org/browse/SERVER-20104</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Environment:&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Stand alone mongod&lt;/li&gt;
	&lt;li&gt;3.1.6v&lt;/li&gt;
	&lt;li&gt;WiredTiger configured with zlib compression and 3GB cache size&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Observation/Issues:&lt;/p&gt;

&lt;ol&gt;
	&lt;li&gt;When running w/ 100 + threads, WT cache reached to 8GB (2.6 x original  cache size).&lt;/li&gt;
&lt;/ol&gt;


&lt;ol&gt;
	&lt;li&gt;When running w/ 9 threads, WT cache reached to 5GB (1.6 x original cache size).&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Breakdown of the 5G memory highlight the followings:&lt;/p&gt;

&lt;p&gt;3 GB in the cache size (as expected)&lt;br/&gt;
1.7 GB - Current_total_thread_cache_bytes (MB) (higher than expectation)&lt;br/&gt;
0.45 GB - Total central_cache_free_bytes  (MB) (higher than expectation)&lt;/p&gt;

&lt;p&gt;Problem: Our memory cache policy set for 1GB free, however the machine accumulate memory above this threshold.  &lt;/p&gt;</description>
                <environment></environment>
        <key id="226893">SERVER-20104</key>
            <summary>WT high memory usage due to high amount of free memory accumulated by TCMalloc</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="backlog-server-execution">Backlog - Storage Execution Team</assignee>
                                    <reporter username="eitan.klein">Eitan Klein</reporter>
                        <labels>
                            <label>32qa</label>
                            <label>WTmem</label>
                    </labels>
                <created>Tue, 25 Aug 2015 02:06:26 +0000</created>
                <updated>Tue, 6 Dec 2022 04:45:38 +0000</updated>
                            <resolved>Thu, 4 Feb 2016 08:10:36 +0000</resolved>
                                                                    <component>Performance</component>
                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>19</watches>
                                                                                                                <comments>
                            <comment id="1163361" author="alexander.gorrod" created="Thu, 4 Feb 2016 08:10:36 +0000"  >&lt;p&gt;This issue is reporting an issue in a third party library (tcmalloc). We have identified a workaround. There is nothing more to do here.&lt;/p&gt;</comment>
                            <comment id="1101548" author="alexander.gorrod" created="Tue, 1 Dec 2015 03:34:41 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bruce.lucas&quot; class=&quot;user-hover&quot; rel=&quot;bruce.lucas&quot;&gt;bruce.lucas&lt;/a&gt; Sorry I sat on this for a long time. I think that the ability to tune this behavior using the &lt;tt&gt;TCMALLOC_AGGRESSIVE_DECOMMIT=true&lt;/tt&gt; environment &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-20104?focusedCommentId=1066474&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-1066474&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;variable&lt;/a&gt; should be enough to satisfy the issue in this ticket.&lt;/p&gt;

&lt;p&gt;That is, if a workload is causing the tcmalloc &lt;tt&gt;thread_cache_free_bytes&lt;/tt&gt; value to grow much larger than the configured 1GB, configure aggressive decommit via an environment variable.&lt;/p&gt;</comment>
                            <comment id="1070066" author="bruce.lucas@10gen.com" created="Mon, 26 Oct 2015 01:23:51 +0000"  >&lt;blockquote&gt;
&lt;p&gt;That is the difference between this ticket and &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-20306&quot; title=&quot;75% excess memory usage under WiredTiger during stress test&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-20306&quot;&gt;&lt;del&gt;SERVER-20306&lt;/del&gt;&lt;/a&gt;. This ticket shows an excessively high value for &lt;tt&gt;thread_cache_free_bytes&lt;/tt&gt;, wheras &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-20306&quot; title=&quot;75% excess memory usage under WiredTiger during stress test&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-20306&quot;&gt;&lt;del&gt;SERVER-20306&lt;/del&gt;&lt;/a&gt; shows a spike in memory consumption (in tcmalloc).&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;That spike takes the form of an excessively high value for &lt;tt&gt;central_cache_free_bytes&lt;/tt&gt;, vs &lt;tt&gt;thread_cache_free_bytes&lt;/tt&gt; for this workload, so in both cases it is a large accumulation of free memory, albeit in different places. On &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-20306&quot; title=&quot;75% excess memory usage under WiredTiger during stress test&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-20306&quot;&gt;&lt;del&gt;SERVER-20306&lt;/del&gt;&lt;/a&gt; at least, based on circumstantial evidence, it appears to be due to fragmentation or the like. Have you ruled out the possibility that the cause of the issue on this ticket is the same, the difference being whether tcmalloc does or does not return the thread cache free memory to the central cache?&lt;/p&gt;</comment>
                            <comment id="1068711" author="nick@innsenroute.com" created="Fri, 23 Oct 2015 04:34:09 +0000"  >&lt;p&gt;You can point the load producer at a Linux-based MongoDb. PM me and I can explain how to set it up.&lt;/p&gt;</comment>
                            <comment id="1068690" author="alexander.gorrod" created="Fri, 23 Oct 2015 03:45:26 +0000"  >&lt;blockquote&gt;&lt;p&gt;Are you seeing similar thread_cache_free_bytes as you see in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-20306&quot; title=&quot;75% excess memory usage under WiredTiger during stress test&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-20306&quot;&gt;&lt;del&gt;SERVER-20306&lt;/del&gt;&lt;/a&gt;?&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;That is the difference between this ticket and &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-20306&quot; title=&quot;75% excess memory usage under WiredTiger during stress test&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-20306&quot;&gt;&lt;del&gt;SERVER-20306&lt;/del&gt;&lt;/a&gt;. This ticket shows an excessively high value for &lt;tt&gt;thread_cache_free_bytes&lt;/tt&gt;, wheras &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-20306&quot; title=&quot;75% excess memory usage under WiredTiger during stress test&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-20306&quot;&gt;&lt;del&gt;SERVER-20306&lt;/del&gt;&lt;/a&gt; shows a spike in memory consumption (in tcmalloc).&lt;/p&gt;

&lt;blockquote&gt;&lt;p&gt;Do you think this is operating system specific? At first glance, I do not believe it is an OS specific issue.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;It is hard to tell. I spent a fair while attempting to reproduce on Linux without any luck. The load-producer is a Windows specific program, so I can&apos;t be sure that I was replicating the same workload. I&apos;m reasonably confident this is specific to Windows.&lt;/p&gt;

&lt;p&gt;&amp;gt; I wonder if the thread free lists grow too large, or something else is going wrong?&lt;/p&gt;

&lt;p&gt;I don&apos;t have an answer to that. Would isolating that down help lead to a fix for this problem? I&apos;m happy to provide access to the machine I&apos;ve been using for testing, if you think we could get more useful information.&lt;/p&gt;</comment>
                            <comment id="1068492" author="mark.benvenuto" created="Thu, 22 Oct 2015 21:54:28 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=alexander.gorrod&quot; class=&quot;user-hover&quot; rel=&quot;alexander.gorrod&quot;&gt;alexander.gorrod&lt;/a&gt; Are you seeing similar &lt;tt&gt;thread_cache_free_bytes&lt;/tt&gt; as you see in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-20306&quot; title=&quot;75% excess memory usage under WiredTiger during stress test&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-20306&quot;&gt;&lt;del&gt;SERVER-20306&lt;/del&gt;&lt;/a&gt;? Do you think this is operating system specific? At first glance, I do not believe it is an OS specific issue. The Windows specific version makes TCMalloc heap private to WiredTiger so all these allocations in the TCMalloc heap are only from WT. I can dig into TCMalloc statistics (&lt;tt&gt;MallocExtension::instance()-&amp;gt;GetStats()&lt;/tt&gt; to see if there is something about our memory allocation patterns that are causing issues. I wonder if the thread free lists grow too large, or something else is going wrong?&lt;/p&gt;

&lt;p&gt;I am planning on modify the server to call GetStats. If you attach GDB, this script may help get detailed information: &lt;a href=&quot;https://gist.github.com/alk/1148755&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://gist.github.com/alk/1148755&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="1067593" author="alexander.gorrod" created="Thu, 22 Oct 2015 05:05:34 +0000"  >&lt;p&gt;I ran the same test with tcmalloc 2.4, and the behavior is not improved. &lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/95153/95153_win_tcmalloc24.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;My current recommendation is to leave this alone in the MongoDB code. It is possible to replicate the aggressive decommit behavior by setting an environment variable:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;TCMALLOC_AGGRESSIVE_DECOMMIT=true&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Upgrading to tcmalloc 2.4 in MongoDB shows a degradation in behavior for this workload.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mark.benvenuto&quot; class=&quot;user-hover&quot; rel=&quot;mark.benvenuto&quot;&gt;mark.benvenuto&lt;/a&gt; Could you review this ticket and let me know if you can think of anything else we could do to alleviate the issue?&lt;/p&gt;</comment>
                            <comment id="1066479" author="alexander.gorrod" created="Wed, 21 Oct 2015 02:36:48 +0000"  >&lt;p&gt;A further resource is that there has been a change in upstream tcmalloc implementation to add a maximum memory budget for tcmalloc:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/gperftools/gperftools/issues/451&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/gperftools/gperftools/issues/451&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The configuration option is TCMALLOC_HEAP_LIMIT_MB. That code hasn&apos;t made it into a release of tcmalloc yet, but it should help with this problem in the future.&lt;/p&gt;</comment>
                            <comment id="1066474" author="alexander.gorrod" created="Wed, 21 Oct 2015 02:17:31 +0000"  >&lt;p&gt;I reproduced this on a Windows 2012 r2, running with the latest 3.2 release candidate I see the same behavior as reported here:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/95006/95006_win_tcmalloc_default.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;You can see that the run went for between 6 to 8 hours. The total thread cache bytes grows to about 3.5GB.&lt;/p&gt;

&lt;p&gt;I then made an adjustment to the tcmalloc configuration to enable aggressive decommit, and it generated the following results:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/95005/95005_win_tcmalloc_agg.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;The run was going for between 12 and 15 hours. You can see that the total thread cache bytes only grows to 1.4GB - which is still above the configured 1GB maximum, but much less worrying than 3.5x the configured maximum.&lt;/p&gt;

&lt;p&gt;I intend to re-run again with a newer release of tcmalloc (2.4, up from 2.2). The more recent release of tcmalloc enables aggressive decommit as the default.&lt;/p&gt;</comment>
                            <comment id="1063593" author="nick@innsenroute.com" created="Sat, 17 Oct 2015 16:18:04 +0000"  >&lt;p&gt;And as a further note, from a practical perspective, this isn&apos;t much of an issue for me, although from a technical perspective I&apos;m sure you want to track it down. A WT caching strategy that addresses the issues with Windows file system cache has a far bigger impact in my world that this (&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-19795&quot; title=&quot;mongod memory consumption higher than WT cache size (at least on Windows 2008 R2)&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-19795&quot;&gt;&lt;del&gt;SERVER-19795&lt;/del&gt;&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-20991&quot; title=&quot;WiredTiger caching strategy needs improvement on Windows&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-20991&quot;&gt;&lt;del&gt;SERVER-20991&lt;/del&gt;&lt;/a&gt;).&lt;/p&gt;</comment>
                            <comment id="1063586" author="nick@innsenroute.com" created="Sat, 17 Oct 2015 16:13:14 +0000"  >&lt;p&gt;A quick note that in some recent tests with 3.2RC0, I only see overage with lower WT caps. For example, setting the cache size to &amp;lt;=4GB, is see an extra ~1.6 - 2 GB of memory usage, but when setting the cache size to 10GB, I don&apos;t see any overage.&lt;/p&gt;</comment>
                            <comment id="1058081" author="nick@innsenroute.com" created="Mon, 12 Oct 2015 16:27:08 +0000"  >&lt;p&gt;@Alexander Gorrod - I can provide you with the repro for this, although 20306 looks like it is probably the same issue and the repro there looks more straight-forward. Email me if you want the repro.&lt;/p&gt;</comment>
                            <comment id="1057588" author="alexander.gorrod" created="Mon, 12 Oct 2015 04:59:48 +0000"  >&lt;p&gt;I haven&apos;t been able to reproduce this behavior by reconstructing the described workload. There is another ticket for a similar memory consumption issue &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-20306&quot; title=&quot;75% excess memory usage under WiredTiger during stress test&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-20306&quot;&gt;&lt;del&gt;SERVER-20306&lt;/del&gt;&lt;/a&gt;. I think we should close this ticket as cannot reproduce. &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ramon.fernandez&quot; class=&quot;user-hover&quot; rel=&quot;ramon.fernandez&quot;&gt;ramon.fernandez&lt;/a&gt; do you agree?&lt;/p&gt;</comment>
                            <comment id="1015183" author="eitan.klein" created="Wed, 26 Aug 2015 17:36:46 +0000"  >&lt;p&gt;More details from Nick J&lt;/p&gt;

&lt;p&gt;Configuration:&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;3.1.7, Stand Alone&lt;/li&gt;
	&lt;li&gt;WT configured for 3 GB of cache.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;~100 threads produced foot print of 8GB heap allocation. but with ~5 GB of free memory in TCMalloc (per thread cache + central cache) &lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/87682/87682_100threads.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;</comment>
                            <comment id="1013513" author="michael.cahill" created="Tue, 25 Aug 2015 02:33:29 +0000"  >&lt;p&gt;It looks like this is really concerning caching within tcmalloc &amp;#8211; from the stats, the WiredTiger cache stays flat at ~2.4GB and tcmalloc reports ~2.5GB memory allocated, so there is no significant amount of additional memory allocated by MongoDB or WiredTiger.&lt;/p&gt;

&lt;p&gt;MongoDB sets tcmalloc.max_total_thread_cache_bytes to 1GB by default &amp;#8211; it seems to be going over that in this case.  &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=acm&quot; class=&quot;user-hover&quot; rel=&quot;acm&quot;&gt;acm&lt;/a&gt;, is there someone on your team who can investigate?&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="228592">SERVER-20306</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="177589">SERVER-16773</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="195160">SERVER-17966</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="87681" name="100threads.html" size="4600703" author="eitan.klein" created="Wed, 26 Aug 2015 17:34:21 +0000"/>
                            <attachment id="87682" name="100threads.png" size="106051" author="eitan.klein" created="Wed, 26 Aug 2015 17:34:21 +0000"/>
                            <attachment id="87394" name="3.1.6-3Gcache.html" size="4656816" author="eitan.klein" created="Tue, 25 Aug 2015 02:06:26 +0000"/>
                            <attachment id="198175" name="buildInfo.txt" size="1313" author="rmaheshwari" created="Tue, 9 Oct 2018 14:57:15 +0000"/>
                            <attachment id="198178" name="collStatsLocalOplog.txt" size="6898" author="rmaheshwari" created="Tue, 9 Oct 2018 14:58:16 +0000"/>
                            <attachment id="198179" name="getCmdLineOpts.txt" size="917" author="rmaheshwari" created="Tue, 9 Oct 2018 14:58:29 +0000"/>
                            <attachment id="198176" name="hostInfo.txt" size="1238" author="rmaheshwari" created="Tue, 9 Oct 2018 14:57:25 +0000"/>
                            <attachment id="198177" name="rsStatus.txt" size="1625" author="rmaheshwari" created="Tue, 9 Oct 2018 14:58:03 +0000"/>
                            <attachment id="198174" name="serverStatus.txt" size="22019" author="rmaheshwari" created="Tue, 9 Oct 2018 14:55:59 +0000"/>
                            <attachment id="95153" name="win_tcmalloc24.png" size="49993" author="alexander.gorrod@mongodb.com" created="Thu, 22 Oct 2015 04:56:10 +0000"/>
                            <attachment id="95005" name="win_tcmalloc_agg.png" size="49152" author="alexander.gorrod@mongodb.com" created="Wed, 21 Oct 2015 02:07:15 +0000"/>
                            <attachment id="95006" name="win_tcmalloc_default.png" size="49618" author="alexander.gorrod@mongodb.com" created="Wed, 21 Oct 2015 02:07:15 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>15.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25136"><![CDATA[Storage Execution]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 25 Aug 2015 02:33:29 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        8 years, 1 week, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>alexander.golin@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            8 years, 1 week, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>alexander.gorrod@mongodb.com</customfieldvalue>
            <customfieldvalue>backlog-server-execution</customfieldvalue>
            <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>eitan.klein</customfieldvalue>
            <customfieldvalue>mark.benvenuto@mongodb.com</customfieldvalue>
            <customfieldvalue>michael.cahill@mongodb.com</customfieldvalue>
            <customfieldvalue>nick@innsenroute.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrkwif:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrek0v:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;mongod --dbpath=d:\mongo --port=27200 --wiredTigerCacheSizeGB=3 --wiredTigerJournalCompressor=zlib --wiredTigerCollectionBlockCompressor=zlib&lt;/p&gt;

&lt;p&gt;User workload (Nick J)&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsfpz3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>