<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:35:19 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-34027] Production issue under high load.</title>
                <link>https://jira.mongodb.org/browse/SERVER-34027</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We had an issue in production yesturday:&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;at ~16:31 UTC undetected queries starts to intensively read cold data. Read form disks greatly increased, as well as eviction from mongo&apos;s page cache. This unexpected load lasts till ~17:12.&lt;br/&gt;
  The load is not issue by itself remarkable for report here. But it leads to following bad behavour:&lt;/li&gt;
	&lt;li&gt;at 16:42:44 mongod literally freezes. It doesn&apos;t respond to any thing, doesn&apos;t write anything to log, doesn&apos;t send statistic till 16:43:06.&lt;br/&gt;
I believe it is bug.&lt;br/&gt;
I&apos;m attaching diagnostic data file that covers that period, and screenshot of monitoring for long period (16:20-17:30) and focused on issue (short 16:40-16:45)&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Environment: aws i3.x16large, mondo data path is placed on lvm volume over two NVMe devices.&lt;/p&gt;</description>
                <environment></environment>
        <key id="514435">SERVER-34027</key>
            <summary>Production issue under high load.</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="david.daly@mongodb.com">David Daly</assignee>
                                    <reporter username="funny.falcon@gmail.com">&#1070;&#1088;&#1080;&#1081; &#1057;&#1086;&#1082;&#1086;&#1083;&#1086;&#1074;</reporter>
                        <labels>
                            <label>former-quick-wins</label>
                    </labels>
                <created>Wed, 21 Mar 2018 11:03:06 +0000</created>
                <updated>Fri, 7 Apr 2023 14:02:39 +0000</updated>
                            <resolved>Sun, 2 Aug 2020 06:22:37 +0000</resolved>
                                    <version>3.6.1</version>
                                                    <component>Stability</component>
                                        <votes>1</votes>
                                    <watches>22</watches>
                                                                                                                <comments>
                            <comment id="2734220" author="henrik.ingo@10gen.com" created="Thu, 16 Jan 2020 10:31:43 +0000"  >&lt;p&gt;Continuing to work on this issue I have also started to focus on the central cache. (This holds free memory objects between the pageheap and per-thread caches.)&lt;/p&gt;

&lt;p&gt;I just wanted to store for the record, the central_cache_free_bytes here is a steady 35 GB. It is not affected by the stall / decommits from pageheap to OS.&lt;/p&gt;

&lt;p&gt; &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/243065/243065_henrik-include-centralcache.png&quot; width=&quot;600&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;In my own tests, under heavy disk bound load the central cache is sometimes as high as 100 GB.&lt;/p&gt;</comment>
                            <comment id="2601851" author="henrik.ingo@10gen.com" created="Thu, 12 Dec 2019 16:04:44 +0000"  >&lt;p&gt;I&apos;ve been working on trying to reproduce this issue. Based on analysing the diagnostic data file you provided, I created a sysbench workload &lt;a href=&quot;https://github.com/mongodb-labs/benchmarks/blob/master/sysbench/lua/tcmalloc_repro.lua&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;tcmalloc_repro.lua&lt;/a&gt;. The commentary at the top of the file summarizes the characteristics I deduced from this ticket.&lt;/p&gt;

&lt;p&gt;Running first the data load for 3 days and the actual workload for 3 hours, I was able to reproduce a situation where tcmalloc pageheap_free_memory reaches 62 - 68 GB, like in this ticket. However, I was NOT able to reproduce an actual stall. &lt;/p&gt;

&lt;p&gt;The difference between the diagnostic data analyzed in this thread vs my repro is that in my repro pageheap decommit count remains small (about 10, vs 3 million above). This means that when tcmalloc finally does decommit memory, it is able to decommit more bytes at a time and therefore free memory becomes available quickly enough that mongod doesn&apos;t get completely stuck. In this ticket average bytes per decommit is 23KB and in my repro it was 54MB. &lt;/p&gt;

&lt;p&gt;In related news &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33296&quot; title=&quot;Excessive memory usage due to heap fragmentation&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33296&quot;&gt;SERVER-33296&lt;/a&gt; is now done. This allows to adjust tcmalloc release rate via setParameter.&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;db.adminCommand({setParameter: 1, tcmallocReleaseRate: 10})&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;This is backported to all stable branches down to 3.6, and will become available in the next stable releases of each.&lt;/p&gt;

&lt;p&gt;From what we have seen so far, we believe that dialing up tcmallocReleaseRate to a bigger value than 1, will prevent pageheap_free_memory from growing indefinitely and that in turn would also avoid stalls such as seen in this ticket. The first part of that I will be able to verify with tcmalloc_repro.lua.&lt;/p&gt;</comment>
                            <comment id="2594088" author="xgen-internal-githook" created="Mon, 9 Dec 2019 16:55:54 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;email&apos;: &apos;henrik.ingo@mongodb.com&apos;, &apos;name&apos;: &apos;Henrik Ingo&apos;, &apos;username&apos;: &apos;henrikingo&apos;}
&lt;p&gt;Message: PERF-1898 Add test_control.yml for &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-34027&quot; title=&quot;Production issue under high load.&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-34027&quot;&gt;&lt;del&gt;SERVER-34027&lt;/del&gt;&lt;/a&gt; repro (#502)&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/10gen/dsi/commit/414e539b42e3a66684548b6a98024fd7c3bf2e91&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/10gen/dsi/commit/414e539b42e3a66684548b6a98024fd7c3bf2e91&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2364131" author="henrik.ingo@10gen.com" created="Fri, 9 Aug 2019 09:25:12 +0000"  >&lt;p&gt;We recently learned that TCMALLOC_RELEASE_RATE can be used to make tcmalloc release memory more frequently to the OS. We haven&apos;t tested anything that would resemble a repro of this issue, but I could speculate that using TCMALLOC_RELEASE_RATE=10 could cause tcmalloc to release memory more frequently and in smaller chunks.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42697&quot; title=&quot;Expose tcmalloc_release_rate via setParameter&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42697&quot;&gt;&lt;del&gt;SERVER-42697&lt;/del&gt;&lt;/a&gt; will add tcmalloc_release_rate tuning via setParameter.&lt;/p&gt;</comment>
                            <comment id="2244701" author="acm" created="Mon, 13 May 2019 16:52:06 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=david.daly&quot; class=&quot;user-hover&quot; rel=&quot;david.daly&quot;&gt;david.daly&lt;/a&gt; - Handing over to you per e-mail discussion.&lt;/p&gt;</comment>
                            <comment id="1970810" author="mark.benvenuto" created="Wed, 8 Aug 2018 20:05:53 +0000"  >&lt;p&gt;The most interesting sample is the sample the was collected from &lt;tt&gt;2018-03-20T16:42:44Z&lt;/tt&gt; to &lt;tt&gt;2018-03-20T16:43:06.122Z&lt;/tt&gt;. This meant that it took approximately ~24 seconds to collect one FTDC sample when it normally takes &amp;lt; 10 ms.&lt;/p&gt;

&lt;p&gt;Accounting for where the time is spent is hard. The serverStatus command usually logs that which sub-section takes the longest (which is probably long gone). What is clear is the following:&lt;/p&gt;

&lt;p&gt;1. The change in decommits between the this sample and previous sample: 3010448 (3 million).&lt;br/&gt;
2. To report the number of decommits or any other TCMalloc stats requires taking the page heap lock (&lt;a href=&quot;https://github.com/mongodb/mongo/blob/210bb5d91cb3c77bb3ed169114f8b85cd1062fb3/src/third_party/gperftools-2.5/src/tcmalloc.cc#L642-L778&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/210bb5d91cb3c77bb3ed169114f8b85cd1062fb3/src/third_party/gperftools-2.5/src/tcmalloc.cc#L642-L778&lt;/a&gt;)&lt;br/&gt;
3. We know that allocate pages takes this same lock (&lt;a href=&quot;https://github.com/mongodb/mongo/blob/210bb5d91cb3c77bb3ed169114f8b85cd1062fb3/src/third_party/gperftools-2.5/src/tcmalloc.cc#L1229-L1258&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/210bb5d91cb3c77bb3ed169114f8b85cd1062fb3/src/third_party/gperftools-2.5/src/tcmalloc.cc#L1229-L1258&lt;/a&gt;) and in order to limit the amount of memory fragmentation, TCMalloc may garbage collect memory (&lt;a href=&quot;https://github.com/mongodb/mongo/blob/210bb5d91cb3c77bb3ed169114f8b85cd1062fb3/src/third_party/gperftools-2.5/src/page_heap.cc#L124-L154&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/210bb5d91cb3c77bb3ed169114f8b85cd1062fb3/src/third_party/gperftools-2.5/src/page_heap.cc#L124-L154&lt;/a&gt;)&lt;br/&gt;
4. We did 71 scavenges during this same period from a Central free list bucket growing too large.&lt;/p&gt;

&lt;p&gt;This meant that is was the TCMalloc page heap GC that stalled the system since calling &lt;tt&gt;munmap&lt;/tt&gt; 3 million times in a spin lock is slow.&lt;/p&gt;

&lt;p&gt;There are a few possible fixes:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Introduce a new state into the free list location enum (&lt;a href=&quot;https://github.com/mongodb/mongo/blob/210bb5d91cb3c77bb3ed169114f8b85cd1062fb3/src/third_party/gperftools-2.5/src/span.h#L64-L66&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/210bb5d91cb3c77bb3ed169114f8b85cd1062fb3/src/third_party/gperftools-2.5/src/span.h#L64-L66&lt;/a&gt;) to support a garbage collect that can be deferred to a separate thread that will do a background free of pages in smaller increments.&lt;/li&gt;
	&lt;li&gt;Temporarily turn on aggressive decommit and then make disable after a period of time instead of flushing all memory. This may not help the fragmentation of the page heap but it would prevent the massive stall.&lt;/li&gt;
	&lt;li&gt;Turn off the fragmentation flush or make it smaller&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Upstream has not made any changes to this code as of the current date. We could pursue filling a bug after creating a smaller repro, and solicit their opinion.&lt;/p&gt;</comment>
                            <comment id="1967213" author="ian@10gen.com" created="Sun, 5 Aug 2018 21:15:33 +0000"  >&lt;p&gt;ping &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mark.benvenuto&quot; class=&quot;user-hover&quot; rel=&quot;mark.benvenuto&quot;&gt;mark.benvenuto&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1840920" author="bruce.lucas@10gen.com" created="Wed, 21 Mar 2018 16:35:42 +0000"  >&lt;p&gt;Hi Yura,&lt;/p&gt;

&lt;p&gt;Thank you for the very detailed problem report! Here&apos;s what we see in the diagnostic data:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/182295/182295_decommit.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;It looks like was caused by tcmalloc decommitting (returning to o/s) its entire pageheap of free memory, consisting of about 66 GB in about 3 M spans. Based on that it appears that this is the same issue as &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-31417&quot; title=&quot;Improve tcmalloc when decommitting large amounts of memory&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-31417&quot;&gt;SERVER-31417&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Bruce&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="441260">SERVER-31417</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="441260">SERVER-31417</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10520">
                    <name>Problem/Incident</name>
                                            <outwardlinks description="causes">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                                        </outwardlinks>
                                                                <inwardlinks description="is related to">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="182255" name="MongoStat-2018-03-21-long.png" size="468018" author="funny_falcon" created="Wed, 21 Mar 2018 10:53:55 +0000"/>
                            <attachment id="182254" name="MongoStat-2018-03-21-short.png" size="400325" author="funny_falcon" created="Wed, 21 Mar 2018 10:53:56 +0000"/>
                            <attachment id="182253" name="SystemStat-2018-03-21-long.png" size="504138" author="funny_falcon" created="Wed, 21 Mar 2018 10:53:55 +0000"/>
                            <attachment id="182252" name="SystemStat-2018-03-21-short.png" size="333226" author="funny_falcon" created="Wed, 21 Mar 2018 10:53:55 +0000"/>
                            <attachment id="182295" name="decommit.png" size="149617" author="bruce.lucas@mongodb.com" created="Wed, 21 Mar 2018 16:34:53 +0000"/>
                            <attachment id="182251" name="diagnostic.tar.gz" size="10534902" author="funny_falcon" created="Wed, 21 Mar 2018 10:54:49 +0000"/>
                            <attachment id="243065" name="henrik-include-centralcache.png" size="171641" author="henrik.ingo@mongodb.com" created="Thu, 16 Jan 2020 10:25:09 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 21 Mar 2018 16:35:42 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 3 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/PERF-1846'>PERF-1846</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 3 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>andrew.morrow@mongodb.com</customfieldvalue>
            <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>david.daly@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>henrik.ingo@mongodb.com</customfieldvalue>
            <customfieldvalue>ian@mongodb.com</customfieldvalue>
            <customfieldvalue>mark.benvenuto@mongodb.com</customfieldvalue>
            <customfieldvalue>funny.falcon@gmail.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htt6rj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr7c1z:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2910">Dev Tools 2019-05-06</customfieldvalue>
    <customfieldvalue id="2942">Dev Tools 2019-04-22</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[bruce.lucas@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htssyv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>