<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:51:19 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-39191] Performance regression for counts post-sharding</title>
                <link>https://jira.mongodb.org/browse/SERVER-39191</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;To filter orphans during counts, sharded clusters avoid COUNT_SCAN in favor of an IXSCAN, and examine documents. Especially for queries that do not include the shard key, this may be difficult to avoid. But, the impact can be noticeable for some workloads.&lt;/p&gt;

&lt;p&gt;Original report follows:&lt;/p&gt;
&lt;hr /&gt;
&lt;p&gt;I have a simple collection with 40M documents distributed ver 2 shards. The documents contain a field named COUNTRY with a sparse index on it.&lt;/p&gt;

&lt;p&gt;I am trying to count documents with a specific value for COUNTRY. When counting on the shards directly the plan looks correct and uses COUNT_SCAN with the index, returns 6M docs on each shard without loading them.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{{2019-01-11T16:03:02.643+0000 I COMMAND [conn102] command segmentation.contacts appName: &quot;MongoDB Shell&quot; command: count { count: &quot;contacts&quot;, query:{ COUNTRY: &quot;US&quot; }, fields: {}, lsid: \{ id: UUID(&quot;60be11b5-6299-416b-9ead-8c58fd3656b8&quot;) }, $clusterTime: { clusterTime: Timestamp(1547221085, 2), signature:{ hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 }}, $db: &quot;segmentation&quot; } planSummary: COUNT_SCAN \{ COUNTRY: 1 } keysExamined:6923922 docsExamined:0 numYields:54093 reslen:340 locks:{ Global: { acquireCount:{ r: 54094 }}, Database: { acquireCount:{ r: 54094 }}, Collection: { acquireCount:{ r: 54094 }} } protocol:op_msg 2747ms}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;Running the exact same query on MONGOS, the count is distributed to the shards but I can see in the logs of each shard that the plan changed and it used IDX_SCAN instead and loaded all the documents which was much slower.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{{2019-01-11T16:04:13.104+0000 I COMMAND [conn111] command segmentation.contacts appName: &quot;MongoDB Shell&quot; command: count { count: &quot;contacts&quot;, query:{ COUNTRY: &quot;US&quot; }, allowImplicitCollectionCreation: false, shardVersion: [ Timestamp(474, 1),ObjectId(&apos;5c37917aa0f162a86b270897&apos;) ], lsid: \{ id: UUID(&quot;9be8fb16-f158-4099-bf25-a3a0c97a33c2&quot;), uid: BinData(0, E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855) }, $clusterTime: { clusterTime: Timestamp(1547222609, 1), signature:{ hash: BinData(0, 1F79735A80E06C35800DE9CDC4E92608B2759F1F), keyId: 6644928741854150685 }}, $client: { application:{ name: &quot;MongoDB Shell&quot; }, driver: \{ name: &quot;MongoDB Internal Client&quot;, version: &quot;4.0.5&quot; }, os: \{ type: &quot;Linux&quot;, name: &quot;Ubuntu&quot;, architecture: &quot;x86_64&quot;, version: &quot;18.04&quot; }, mongos: \{ host: &quot;ip-192-168-169-237:27019&quot;, client: &quot;127.0.0.1:59430&quot;, version: &quot;4.0.5&quot; } }, $configServerState: { opTime:{ ts: Timestamp(1547222609, 1), t: 1 }}, $db: &quot;segmentation&quot; } planSummary: IXSCAN \{ COUNTRY: 1 } keysExamined:6923921 docsExamined:6923921 numYields:54093 reslen:320 locks:{ Global: { acquireCount:{ r: 54094 }}, Database: { acquireCount:{ r: 54094 }}, Collection: { acquireCount:{ r: 54094 }} } protocol:op_msg 34784ms}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;Can someone explain me why the plan changed or why the docs have to be scanned ? The performance impact is huge...&lt;/p&gt;

&lt;p&gt;Attached execution plan from MONGOS. This issue was also referenced in &lt;a href=&quot;https://dba.stackexchange.com/questions/226936/mongos-uselessly-scanning-indexed-documents-when-counting-on-shards?noredirect=1#comment447171_226936&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;StackExchange&#160;&lt;/a&gt;&lt;/p&gt;</description>
                <environment></environment>
        <key id="677906">SERVER-39191</key>
            <summary>Performance regression for counts post-sharding</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="ian.boros@mongodb.com">Ian Boros</assignee>
                                    <reporter username="jmoutte">Julien Moutte</reporter>
                        <labels>
                            <label>qexec-team</label>
                    </labels>
                <created>Fri, 25 Jan 2019 09:15:23 +0000</created>
                <updated>Sun, 29 Oct 2023 22:24:47 +0000</updated>
                            <resolved>Thu, 15 Oct 2020 20:53:46 +0000</resolved>
                                    <version>4.0.5</version>
                                    <fixVersion>4.9.0</fixVersion>
                                    <component>Sharding</component>
                                        <votes>2</votes>
                                    <watches>25</watches>
                                                                                                                <comments>
                            <comment id="3446535" author="xgen-internal-githook" created="Thu, 15 Oct 2020 16:31:18 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Sebastian Mayr&apos;, &apos;email&apos;: &apos;smayr@atlassian.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-39191&quot; title=&quot;Performance regression for counts post-sharding&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-39191&quot;&gt;&lt;del&gt;SERVER-39191&lt;/del&gt;&lt;/a&gt; Optimize count queries on sharded collections&lt;/p&gt;

&lt;p&gt;When counts were made orphan-aware the addition of the SHARD_FILTER&lt;br/&gt;
stage started preventing the COUNT_SCAN optimization on all sharded&lt;br/&gt;
collections, leading to significant performance loss on some workloads.&lt;/p&gt;

&lt;p&gt;However, there can not be any orphans if the shard key is fully&lt;br/&gt;
specified: A mongod will never have orphans for chunk ranges&lt;br/&gt;
it owns.&lt;/p&gt;

&lt;p&gt;This patch changes the query system to omit the SHARDING_FILTER stage in cases&lt;br/&gt;
where the query specifies the full shard key, regaining lost performance.&lt;/p&gt;

&lt;p&gt;Closes #1369.&lt;/p&gt;

&lt;p&gt;Signed-off-by: Ian Boros &amp;lt;ian.boros@mongodb.com&amp;gt;&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/14bfbd8833706912617c0c904cce9847fbbbd0f1&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/14bfbd8833706912617c0c904cce9847fbbbd0f1&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3362420" author="sebmaster" created="Thu, 27 Aug 2020 16:12:37 +0000"  >&lt;p&gt;I opened a PR with what I think should be the correct changes on GH, but it probably needs a bit of attention from a Mongo team member: &lt;a href=&quot;https://github.com/mongodb/mongo/pull/1369&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/pull/1369&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3362374" author="garaudy.etienne" created="Thu, 27 Aug 2020 15:55:38 +0000"  >&lt;p&gt;Based on &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sebmaster&quot; class=&quot;user-hover&quot; rel=&quot;sebmaster&quot;&gt;sebmaster&lt;/a&gt;&apos;s suggestion, query seems best equipped to make the right optimizations here. &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kateryna.kamenieva&quot; class=&quot;user-hover&quot; rel=&quot;kateryna.kamenieva&quot;&gt;kateryna.kamenieva&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3193267" author="sebmaster" created="Fri, 5 Jun 2020 20:53:17 +0000"  >&lt;p&gt;We&apos;ve encountered this issue quite recently as well.&lt;/p&gt;

&lt;p&gt;We had the following plan to lessen the impact of this considerably in one special case: Counting with a fully specified $eq shard key. The idea is that in those cases, mongos would only send the count request to a single shard that is responsible for the respective chunk. This should guarantee that the shard cannot have orphans matching the query.&lt;/p&gt;

&lt;p&gt;Could someone check our thinking on that? If this is correct, I have some code + tests already written up that strips the sharding filter in this case that we&apos;d like to contribute.&lt;/p&gt;</comment>
                            <comment id="2966922" author="asya" created="Wed, 11 Mar 2020 15:47:22 +0000"  >&lt;p&gt;Note that sharding filter still has to be applied, but it can be done from an index, if the index covers query field PLUS shard filter field(s).&lt;/p&gt;</comment>
                            <comment id="2966909" author="asya" created="Wed, 11 Mar 2020 15:45:47 +0000"  >&lt;p&gt;I believe this is (somewhat) a duplicate of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-39241&quot; title=&quot;Plan scoring incorrectly applies noFetchBonus bonus to all plans when no projection is provided&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-39241&quot;&gt;&lt;del&gt;SERVER-39241&lt;/del&gt;&lt;/a&gt; which is fixed in 4.3.2.&lt;/p&gt;</comment>
                            <comment id="2134474" author="jmoutte" created="Fri, 1 Feb 2019 09:06:00 +0000"  >&lt;p&gt;Sure, let&apos;s try.&lt;/p&gt;

&lt;p&gt;Our use case is basically a graphical query builder where users can add filtering criteria progressively to segment a big data collection. They start with the whole data collection and they see the total count or a count of distinct values on a specific indexed field, they can then incrementally add filtering criteria and see the size of each segment as they build that segmentation model. The count helps building the model and it should come back fast. Accuracy does not have to be 100%, counting distinct entries with hyperloglog is a possibility.&lt;/p&gt;

&lt;p&gt;For example you could be querying a massive events table from IOT devices belonging to users and you would want to incrementally segment that user base based on criteria applied to that events collection. Hope this is clear enough.&lt;/p&gt;</comment>
                            <comment id="2134040" author="alyson.cabral" created="Thu, 31 Jan 2019 21:25:33 +0000"  >&lt;p&gt;Thanks, Julien for the info!&lt;/p&gt;

&lt;p&gt;Without going into the specific&#160;intricacies of your data (if possible), can you describe what value counts bring to your user-facing application?&#160;Having a well-rounded understanding of use cases for count usage in applications will really help us prioritize this work appropriately.&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2133049" author="jmoutte" created="Thu, 31 Jan 2019 08:23:14 +0000"  >&lt;p&gt;Thanks Eric for the feedback.&lt;/p&gt;

&lt;p&gt;This is a POC to evaluate MONGO for a Big Data segmentation use case:&lt;/p&gt;

&lt;p&gt;Given a massive collection of documents with proper indices we want to count matching documents based on a set of criteria. Sharding is used to distribute both storage and workload in a scale-out scenario.&lt;/p&gt;

&lt;p&gt;Initial tests were done on a non sharded database and the query execution time was okayish (2.7s) for a simple count on an indexed field.&lt;/p&gt;

&lt;p&gt;We were expecting that sharding would add 2 extra qualities :&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;a linear execution time reduction by adding shards and splitting the data / counting.&lt;/li&gt;
	&lt;li&gt;the possibility to handle data growth while maintaining execution time stable.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;The opposite happens. When querying shards through mongos, the same query runs in (34s) as documents are examined for this orphaned documents counting. One could argue that quality 2) is somehow still there but the execution time is way too long for our user interface.&lt;/p&gt;

&lt;p&gt;I understand the reason why this happens, so we should change the issue type to improvement instead of bug.&lt;/p&gt;</comment>
                            <comment id="2131174" author="eric.sedor" created="Tue, 29 Jan 2019 21:00:44 +0000"  >&lt;p&gt;Hi Julien, thanks for your patience.&lt;/p&gt;

&lt;p&gt;To accurately count documents in a sharded environment, shards must examine them. This is working as designed as of&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-3645&quot; title=&quot;Sharded collection counts (on primary) can report too many results&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-3645&quot;&gt;&lt;del&gt;SERVER-3645&lt;/del&gt;&lt;/a&gt;&#160;and is necessary to avoid counting&#160;&lt;a href=&quot;https://docs.mongodb.com/manual/reference/glossary/#term-orphaned-document&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;orphaned documents&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;While it may one day be possible to do this without examining the documents directly, it is unlikely this would be able to be changed soon.&lt;/p&gt;

&lt;p&gt;That said, we&apos;d like to ask for more information to help us consider this problem.&lt;/p&gt;

&lt;p&gt;What purpose are the counts serving in your application, are there strict SLAs around the counts in particular, and exactly how much slower did they get for you?&lt;/p&gt;

&lt;p&gt;In the event overall performance has been impacted by the memory requirement of these counts, can you classify the total impact?&lt;/p&gt;

&lt;p&gt;Thank you in advance!&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10520">
                    <name>Problem/Incident</name>
                                            <outwardlinks description="causes">
                                        <issuelink>
            <issuekey id="2367531">SERVER-78071</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is caused by">
                                        <issuelink>
            <issuekey id="21174">SERVER-3645</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="679378">SERVER-39241</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="270985">SERVER-23017</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="1469386">SERVER-50857</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="207174" name="mongos_plan.txt" size="12806" author="jmoutte" created="Fri, 25 Jan 2019 09:11:08 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>10.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[5002K00000j0tk7QAA, 5002K00000kriSrQAI, 5002K00000nBJcHQAW, 5002K00000pkaeEQAQ]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 25 Jan 2019 19:24:29 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 16 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 16 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>alyson.cabral@mongodb.com</customfieldvalue>
            <customfieldvalue>asya.kamsky@mongodb.com</customfieldvalue>
            <customfieldvalue>eric.sedor@mongodb.com</customfieldvalue>
            <customfieldvalue>garaudy.etienne@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>ian.boros@mongodb.com</customfieldvalue>
            <customfieldvalue>jmoutte</customfieldvalue>
            <customfieldvalue>sebmaster</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|huk893:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hua4pj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3616">Query 2020-10-05</customfieldvalue>
    <customfieldvalue id="3617">Query 2020-10-19</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[eric.sedor@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hujuif:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>