<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:42:49 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-36346] Large memory consumption per pending getmore</title>
                <link>https://jira.mongodb.org/browse/SERVER-36346</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;For getmores we allocate a 32 MB buffer for the response up front, which remains allocated while the getmore is pending. (This is the 16 MB bson maximum document size, plus some amount for the response message overhead, rounded up to 32 MB by the buffer allocator). This can add up to a lot of memory overhead if there are a large number of pending getmores, for example if you have a lot of oplog tails or change streams.&lt;/p&gt;

&lt;p&gt;Can this be improved? For example maybe we don&apos;t need to allocate for the worst case up front? Even the simple expedient of teaching the buffer allocator that it doesn&apos;t always need to stick to powers of two would improve this by a factor of two.&lt;/p&gt;

</description>
                <environment></environment>
        <key id="579468">SERVER-36346</key>
            <summary>Large memory consumption per pending getmore</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="mohammad.dashti@mongodb.com">Mohammad Dashti</assignee>
                                    <reporter username="bruce.lucas@mongodb.com">Bruce Lucas</reporter>
                        <labels>
                            <label>query-director-triage</label>
                    </labels>
                <created>Mon, 30 Jul 2018 13:21:50 +0000</created>
                <updated>Sun, 29 Oct 2023 22:29:27 +0000</updated>
                            <resolved>Tue, 26 Oct 2021 16:51:00 +0000</resolved>
                                                    <fixVersion>5.2.0</fixVersion>
                                                        <votes>4</votes>
                                    <watches>43</watches>
                                                                                                                <comments>
                            <comment id="4172056" author="JIRAUSER1259138" created="Fri, 5 Nov 2021 16:58:25 +0000"  >&lt;blockquote&gt;&lt;p&gt;In HELP-7195 customer uses Meteor framework to accomplish something similar to change streams. They were watching 800 collections for updates this way, resulting in 27 GB of memory allocated for the getmores.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;&lt;tt&gt;800 * 32MB = 25.6 GB&lt;/tt&gt; (out of the reported &lt;tt&gt;27 GB&lt;/tt&gt;) should&apos;ve been related to the allocated buffer for pending &lt;tt&gt;getMores&lt;/tt&gt; on tailable cursors. These buffers were likely retained for an extensive period of time waiting for a change to happen and reporting them back to the user. Now, with the change in this ticket (and all related tickets), we take a maximum of &lt;tt&gt;16MB&lt;/tt&gt; buffers for a very limited amount of time (i.e., after having at least a single record to return until returning it back to the user. It&apos;s usually sub-second).&lt;/p&gt;

&lt;p&gt;Basically, for &lt;tt&gt;N&lt;/tt&gt; pending &lt;tt&gt;getMore&lt;/tt&gt; requests on a tailable cursor, we were allocating &lt;tt&gt;N*32MB&lt;/tt&gt; for the whole period until a change occurs on that collection. Then, assuming that we&apos;re using a framework like &lt;tt&gt;Meteor&lt;/tt&gt;, it&apos;ll constantly keep these &lt;tt&gt;N&lt;/tt&gt; cursors, causing a constant &lt;tt&gt;N*32MB&lt;/tt&gt; reserved. With the change in this ticket, now almost nothing will be reserved.&lt;/p&gt;</comment>
                            <comment id="4148884" author="JIRAUSER1259138" created="Tue, 26 Oct 2021 16:50:27 +0000"  >&lt;p&gt;Author:&lt;/p&gt;

{&apos;name&apos;: &apos;Mohammad Dashti&apos;, &apos;email&apos;: &apos;mdashti@gmail.com&apos;, &apos;username&apos;: &apos;mdashti&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-60138&quot; title=&quot;Improve the allocated memory for getMore commands&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-60138&quot;&gt;&lt;del&gt;SERVER-60138&lt;/del&gt;&lt;/a&gt; Improved the allocated memory for the `getMore` commands.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/f6d975343b9f919b61e6d1b4e7333fe0ecfe50c4&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/f6d975343b9f919b61e6d1b4e7333fe0ecfe50c4&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3093651" author="bruce.lucas@10gen.com" created="Mon, 18 May 2020 19:37:40 +0000"  >&lt;p&gt;Tip: it&apos;s much easier to make comparisons if the two runs are plotted on the same charts. If they are separated by a subsantial gap in time you can use the timeline &quot;condensed&quot; option to remove the gap so they both occupy about half of the width.&lt;/p&gt;

&lt;p&gt;Left is new, without prealloc; right is old, with prealloc&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/261394/261394_comparison.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;Generally speaking, this is a very light load with very little concurrency. A heavier load with more concurrency may or may not look different.&lt;/p&gt;


&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                        left, new            right, old&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                        no  prealloc         w/ prealloc&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;read latency            1.5-2x higher&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;scavenge, decommit                           higher, as expected&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;insegs                  higher (!?)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;retrans                                      higher (but insignificant)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;ul&gt;
	&lt;li&gt;Read latency as measured at the server is higher without prealloc. However the timer for read latency begins after we have allocated the 32 MB response buffer, so the &quot;old&quot; numbers don&apos;t include the time required for allocating the buffer whereas the &quot;new&quot; numbers do, so this could be misleading. What did you see for latency at the client?&lt;/li&gt;
&lt;/ul&gt;


&lt;ul&gt;
	&lt;li&gt;Scavenge and decommit rates are better without prealloc. This is expected as tcmalloc drives its rate of scavenging based on the rate at which bytes are returned to the pageheap, and the huge buffers we&apos;re allocating for getmores can be a significant driver of that and can cause notable peformance problems in heavier loads. So this is good, although this test was not stressful enough to see any improvement for the application as a result of this.&lt;/li&gt;
&lt;/ul&gt;


&lt;ul&gt;
	&lt;li&gt;TCP input segments is higher without the change. Not really sure why. Maybe just due to artificial nature of test?&lt;/li&gt;
&lt;/ul&gt;


&lt;ul&gt;
	&lt;li&gt;Retransmits are lower with the change, though not a particularly significant number to begin with. Also maybe just an artifact of the synthetic test.&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="3044258" author="redbeard0531" created="Fri, 17 Apr 2020 12:34:23 +0000"  >&lt;p&gt;In addition to tailable, we should also consider whether there is any batchSize restriction in the getmore request. It is also wasteful to allocate the giant buffer when the client is asking for only a few documents at a time.&lt;/p&gt;

&lt;p&gt;Additionally, we should probably add a special case to &lt;a href=&quot;https://github.com/mongodb/mongo/blob/05eb5a55e9a7f50db04c0a7a12e6cd06248bbab5/src/mongo/bson/util/builder.h#L404-L416&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/05eb5a55e9a7f50db04c0a7a12e6cd06248bbab5/src/mongo/bson/util/builder.h#L404-L416&lt;/a&gt;, so that whenever we would allocate 16MB, we allocate 16MB + ~64KB instead. The 0.5% overallocation will frequently avoid the 99.5% overallocation we do today, which seems like a good tradeoff. Also, while we are in there, it is probably worth downsizing the allocation size by &lt;tt&gt;sizeof(SharedBuffer::Holder)&lt;/tt&gt; (as long as it is still &amp;gt; minSize) to counteract the extra space requested at &lt;a href=&quot;https://github.com/mongodb/mongo/blob/master/src/mongo/util/shared_buffer.h#L70&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/master/src/mongo/util/shared_buffer.h#L70&lt;/a&gt;, which is probably bumping us up to a higher TCMalloc size class.&lt;/p&gt;</comment>
                            <comment id="1974407" author="charlie.swanson" created="Mon, 13 Aug 2018 18:02:31 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=david.storch&quot; class=&quot;user-hover&quot; rel=&quot;david.storch&quot;&gt;david.storch&lt;/a&gt; I&apos;d actually like to nominate this ticket for quick wins. After investigating a little more, it looks like this should be pretty doable.&lt;/p&gt;

&lt;p&gt;It looks like the getMore command overrides &lt;a href=&quot;https://github.com/mongodb/mongo/blob/e635f4e9552ed3362399850bd993518e7e87b2c6/src/mongo/db/commands/getmore_cmd.cpp#L550&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;Command::reserveBytesForReply()&lt;/tt&gt;&lt;/a&gt; to specify that we should reserve ~16MB. Upon a first glance it looks like we could move this method to the &apos;parsed&apos; &apos;Invocation&apos; of the class, which has access to a GetMoreRequest. I &lt;em&gt;think&lt;/em&gt; there&apos;s only one call site of &apos;reserveBytesForReply&apos;, from &lt;a href=&quot;https://github.com/mongodb/mongo/blob/71df8736a95df0738470e46c98fa8199abb23b4c/src/mongo/db/service_entry_point_common.cpp#L503&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;service_entry_point_common.cpp&lt;/a&gt;. If we have the GetMoreRequest, we could conditionalize the number of bytes to preallocate based on whether the cursor is tailable. If the cursor is tailable we usually wouldn&apos;t expect to need 16MB, since we&apos;ll only be getting the newest data. Does that sound crazy? It seems like it would certainly be a win for a change stream which I&apos;d expect to usually require fewer than 16MB. For other tailable cursors it seems possible that there are a substantial number of full 16MB batches before the tailable part, but I&apos;m not sure what better heuristic we could ever use, or how noticeable a regression like that would be.&lt;/p&gt;

&lt;p&gt;Moving this back to &apos;Needs Triage&apos; to discuss.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="1880248">SERVER-60137</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1880261">SERVER-60138</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                                        </outwardlinks>
                                                                <inwardlinks description="is related to">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="261394" name="comparison.png" size="206112" author="bruce.lucas@mongodb.com" created="Mon, 18 May 2020 19:36:07 +0000"/>
                            <attachment id="260549" name="diagnostic.data-prealloc.zip" size="159853" author="bernard.gorman@mongodb.com" created="Thu, 14 May 2020 11:51:15 +0000"/>
                            <attachment id="260550" name="diagnostic.data.no-prealloc.zip" size="160141" author="bernard.gorman@mongodb.com" created="Thu, 14 May 2020 11:51:13 +0000"/>
                            <attachment id="260551" name="no-prealloc-memory-usage.png" size="585245" author="bernard.gorman@mongodb.com" created="Thu, 14 May 2020 11:51:15 +0000"/>
                            <attachment id="260552" name="prealloc-memory-usage.png" size="584161" author="bernard.gorman@mongodb.com" created="Thu, 14 May 2020 11:51:16 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>11.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[5002K00000cuf1XQAQ, 5002K00000iMo1GQAS, 5002K00000iQZFoQAO, 5002K00000msVmaQAE, 5002K00000nC0uIQAS, 5002K00000ko1zUQAQ, 5002K00000tR7l3QAC, 5002K00000xFdieQAC, 5006R00001qlROuQAM]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 30 Jul 2018 15:04:57 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 13 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/PERF-2565'>PERF-2565</a></s>, <s><a href='https://jira.mongodb.org/browse/PERF-2566'>PERF-2566</a></s>, <s><a href='https://jira.mongodb.org/browse/SERVER-60137'>SERVER-60137</a></s>, <s><a href='https://jira.mongodb.org/browse/SERVER-60138'>SERVER-60138</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 13 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>135.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>charlie.swanson@mongodb.com</customfieldvalue>
            <customfieldvalue>mathias@mongodb.com</customfieldvalue>
            <customfieldvalue>mohammad.dashti@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hu3zjz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr2ptr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3494">Query 2020-03-09</customfieldvalue>
    <customfieldvalue id="3496">Query 2020-03-23</customfieldvalue>
    <customfieldvalue id="3497">Query 2020-04-06</customfieldvalue>
    <customfieldvalue id="3498">Query 2020-04-20</customfieldvalue>
    <customfieldvalue id="3499">Query 2020-05-04</customfieldvalue>
    <customfieldvalue id="3500">Query 2020-05-18</customfieldvalue>
    <customfieldvalue id="3501">Query 2020-06-01</customfieldvalue>
    <customfieldvalue id="4719">QE 2021-09-20</customfieldvalue>
    <customfieldvalue id="5265">QE 2021-10-04</customfieldvalue>
    <customfieldvalue id="5267">QE 2021-10-18</customfieldvalue>
    <customfieldvalue id="5269">QE 2021-11-01</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hu3ltb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>