<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:06:47 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-44725] Audit usages of epoch in sharding code</title>
                <link>https://jira.mongodb.org/browse/SERVER-44725</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;In several places in sharding we use a change of epoch to indicate that a collection had been dropped and recreated. After the refine shard key project however, this logic is no longer valid, since refining a shard key can cause an epoch change. We should audit and fix the places where epoch is used instead of collection UUID to check for collection drops and recreates.&lt;/p&gt;

&lt;p&gt;Specific examples include the MigrationDestinationManager and the CollectionShardingRuntime.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1010199">SERVER-44725</key>
            <summary>Audit usages of epoch in sharding code</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="jack.mulrow@mongodb.com">Jack Mulrow</assignee>
                                    <reporter username="matthew.saltz@mongodb.com">Matthew Saltz</reporter>
                        <labels>
                    </labels>
                <created>Mon, 18 Nov 2019 22:31:05 +0000</created>
                <updated>Sun, 29 Oct 2023 22:14:47 +0000</updated>
                            <resolved>Mon, 24 Feb 2020 19:49:47 +0000</resolved>
                                                    <fixVersion>4.3.4</fixVersion>
                                    <component>Sharding</component>
                                        <votes>0</votes>
                                    <watches>3</watches>
                                                                                                                <comments>
                            <comment id="2905756" author="jack.mulrow" created="Mon, 24 Feb 2020 19:49:19 +0000"  >&lt;p&gt;I don&apos;t think it should be that much work to address either 1) or 2), so I filed &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-46370&quot; title=&quot;Correctly maintain receiving chunks list after shard key refine&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-46370&quot;&gt;&lt;del&gt;SERVER-46370&lt;/del&gt;&lt;/a&gt; to track both. I also filed &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-46371&quot; title=&quot;Error messages and logging should not assume collection was dropped and recreated on epoch change&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-46371&quot;&gt;&lt;del&gt;SERVER-46371&lt;/del&gt;&lt;/a&gt; to track fixing the log lines.&lt;/p&gt;</comment>
                            <comment id="2905339" author="esha.maharishi@10gen.com" created="Mon, 24 Feb 2020 17:54:36 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jack.mulrow&quot; class=&quot;user-hover&quot; rel=&quot;jack.mulrow&quot;&gt;jack.mulrow&lt;/a&gt; what you said makes sense to me.&lt;/p&gt;

&lt;p&gt;I think it&apos;s worth trying to make refineShardKey work correctly when the resumable range deleter is disabled if it is a relatively small amount of effort. A few questions:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;(Bullet 4.1 above) Would it be difficult to fix &lt;tt&gt;rangeOverlapsChunk&lt;/tt&gt; to work correctly if the shard key has been refined?&lt;/li&gt;
	&lt;li&gt;(Bullet 4.2 above) Would it be difficult to make the MigrationDestinationManager store the collection UUID at the beginning of the receive, and change that comparison to be on UUID?&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;I also think it would be good to fix the log lines that are now incorrect, and to update any comments that you found that are outdated.&lt;/p&gt;</comment>
                            <comment id="2877935" author="jack.mulrow" created="Fri, 14 Feb 2020 02:25:46 +0000"  >&lt;p&gt;I grepped the code base for `StaleEpoch`, `.epoch()`, and just `epoch` and here&apos;s an overview of what I found from looking through the matches that looked related:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;We mostly check epochs to detect when routing information was stale and abort operations on mismatches, sometimes retrying automatically after a refresh. Every place I saw that did this used epochs correctly as far as I could tell, and at worst mismatches lead to increased latency or a manual retry from a client. Some specific examples:
	&lt;ol&gt;
		&lt;li&gt;An epoch mismatch for a CRUD op triggers a mongos and shard refresh and an automatic retry &lt;a href=&quot;https://github.com/mongodb/mongo/blob/0d083cfc66abb8804ca5b80c8b2cea534434099b/src/mongo/db/s/collection_sharding_runtime.cpp#L380-L383&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;by throwing a stale config exception&lt;/a&gt;.&lt;/li&gt;
		&lt;li&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/149aae77fd00cbb0d5760881e76eae631e1f0e11/src/mongo/db/s/config/sharding_catalog_manager_chunk_operations.cpp#L831-L840&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;move&lt;/a&gt;, &lt;a href=&quot;https://github.com/mongodb/mongo/blob/0d083cfc66abb8804ca5b80c8b2cea534434099b/src/mongo/db/s/config/sharding_catalog_manager_chunk_operations.cpp#L398-L404&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;split&lt;/a&gt;, and &lt;a href=&quot;https://github.com/mongodb/mongo/blob/0d083cfc66abb8804ca5b80c8b2cea534434099b/src/mongo/db/s/config/sharding_catalog_manager_chunk_operations.cpp#L655-L659&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;merge&lt;/a&gt; chunk all check the epoch before committing, aborting without retrying if there&apos;s a mismatch. The donor in a move also checks its epoch matches the epoch sent by the config server at the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/149aae77fd/src/mongo/db/s/migration_source_manager.cpp#L213-L218&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;beginning of the move&lt;/a&gt; and &lt;a href=&quot;https://github.com/mongodb/mongo/blob/149aae77fd/src/mongo/db/s/migration_source_manager.cpp#L660-L666&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;periodically&lt;/a&gt;.&lt;/li&gt;
		&lt;li&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/149aae77fd/src/mongo/db/s/config/configsvr_refine_collection_shard_key_command.cpp#L116-L120&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;refineCollectionShardKey&lt;/a&gt; and &lt;a href=&quot;https://github.com/mongodb/mongo/blob/0d083cfc66abb8804ca5b80c8b2cea534434099b/src/mongo/db/s/config/configsvr_clear_jumbo_flag_command.cpp#L90-L94&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;clearJumboFlag&lt;/a&gt; are forwarded to the config primary with an epoch and will abort without retrying on an epoch mismatch.&lt;/li&gt;
		&lt;li&gt;Aggregation stages that perform writes ($out and $merge) &lt;a href=&quot;https://github.com/mongodb/mongo/blob/66002c604a9a2cd9c419bad318db0252f576dbd8/src/mongo/s/write_ops/chunk_manager_targeter.cpp#L370-L372&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;use a cached epoch to check&lt;/a&gt; if the shard key for the output collection has changed since the aggregation was planned. Since a the shard key changes after a refine, this seems to work as intended.&lt;/li&gt;
	&lt;/ol&gt;
	&lt;/li&gt;
	&lt;li&gt;There are some miscellaneous places we use epochs that aren&apos;t necessarily related to retrying, but don&apos;t seem incorrect, e.g.
	&lt;ol&gt;
		&lt;li&gt;The map of writes and shard endpoints used in the mongos batch write path &lt;a href=&quot;https://github.com/mongodb/mongo/blob/149aae77fd/src/mongo/s/write_ops/batch_write_op.cpp#L902-L905&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;considers epochs when differentiating endpoints&lt;/a&gt;. It shouldn&apos;t be possible for a cached routing table to have both the pre and post refine epoch for a collection, so I don&apos;t think this is a problem.&lt;/li&gt;
	&lt;/ol&gt;
	&lt;/li&gt;
	&lt;li&gt;There are a few places where we log or return a misleading error message when we detect an epoch change, typically saying the collection &quot;has been dropped and recreated&quot; which isn&apos;t always correct now (e.g. &lt;a href=&quot;https://github.com/mongodb/mongo/blob/0d083cfc66abb8804ca5b80c8b2cea534434099b/src/mongo/db/s/config/sharding_catalog_manager_chunk_operations.cpp#L1136-L1137&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;). We should probably fix these, but I don&apos;t think there&apos;s much urgency. I also found at least one &lt;a href=&quot;https://github.com/mongodb/mongo/blob/16f001e636/src/mongo/db/s/metadata_manager.h#L255-L256&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;out of data comment&lt;/a&gt;.&lt;/li&gt;
	&lt;li&gt;The only logic bugs I found were around maintaining the list of chunk ranges being received by the MetadataManager, that can probably lead to orphans in rare cases, but I think we already knew this. In particular, I saw two problems:
	&lt;ol&gt;
		&lt;li&gt;When setting new filtering metadata in the MetadataManager, we clear entries&#160;from the receiving chunks list &lt;a href=&quot;https://github.com/mongodb/mongo/blob/2bf81c1377b893511179f05f89194a777df450f4/src/mongo/db/s/metadata_manager.cpp#L191&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;that overlap with the new metadata&lt;/a&gt;. This comparison goes through the ChunkManager, which uses key strings to compare ranges,&#160;which doesn&apos;t work correctly for ranges with different numbers of fields, so we might not be removing chunks from this list after a refine.&lt;/li&gt;
		&lt;li&gt;In MigrationDestinationManager::_forgetReceive(), we don&apos;t remove a chunk from the receiving range list &lt;a href=&quot;https://github.com/mongodb/mongo/blob/9aa9fa4a2844f0fe7890e01d621960a0c64607f6/src/mongo/db/s/migration_destination_manager.cpp#L1353&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;if the epoch has changed&lt;/a&gt;. If there&apos;s a refine during the migration, this won&apos;t be correct and we may fail to clear the received range from receiving chunks. There&apos;s a similar check in MSM::_notePending() (that Matthew already linked above), but at that point nothing has been added to the receiving chunks list, so aborting shouldn&apos;t be a problem.&lt;/li&gt;
	&lt;/ol&gt;
	&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;There are a few things to note about the &quot;bugs&quot; in item 4) that make me question how important it is to fix them:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;They seem to require a refine to happen during a migration, which normally won&apos;t happen because a &lt;a href=&quot;https://github.com/mongodb/mongo/blob/0d083cfc66abb8804ca5b80c8b2cea534434099b/src/mongo/db/s/config/configsvr_refine_collection_shard_key_command.cpp#L88-L92&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;refine&lt;/a&gt; and &lt;a href=&quot;https://github.com/mongodb/mongo/blob/0d083cfc66abb8804ca5b80c8b2cea534434099b/src/mongo/db/s/balancer/migration_manager.cpp#L503-L509&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;move chunk&lt;/a&gt; both take a collection distributed lock, so the lock would need to be overtaken or otherwise fail to serialize the operations.&lt;/li&gt;
	&lt;li&gt;The receiving chunks list only seems to be used when the resumable range deleter is disabled. The only exception is &lt;a href=&quot;https://github.com/mongodb/mongo/blob/0d083cfc66abb8804ca5b80c8b2cea534434099b/src/mongo/db/s/cleanup_orphaned_cmd.cpp#L104&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;cleanupOrphaned&lt;/a&gt;, but I believe&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-44160&quot; title=&quot;Make cleanupOrphaned just wait for all range deletion tasks on the ns to finish in FCV 4.4 when &amp;#39;disableResumableRangeDeleter&amp;#39; is false&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-44160&quot;&gt;&lt;del&gt;SERVER-44160&lt;/del&gt;&lt;/a&gt; is intended to fix this already.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;So overall, other than edge cases leaving extra orphans in the old range deletion protocol, everywhere we use epochs seems to work fine if the epoch changes because of a refine instead of a drop/recreate.&#160;We may want to fix up some error messages / logging, but I don&apos;t think there&apos;s work here that needs to block an RC or 4.4.0.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=esha.maharishi&quot; class=&quot;user-hover&quot; rel=&quot;esha.maharishi&quot;&gt;esha.maharishi&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=matthew.saltz&quot; class=&quot;user-hover&quot; rel=&quot;matthew.saltz&quot;&gt;matthew.saltz&lt;/a&gt;, what do you guys think?&lt;/p&gt;</comment>
                            <comment id="2549252" author="esha.maharishi@10gen.com" created="Tue, 19 Nov 2019 16:02:43 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kaloian.manassiev&quot; class=&quot;user-hover&quot; rel=&quot;kaloian.manassiev&quot;&gt;kaloian.manassiev&lt;/a&gt;, note that we should not get rid of epochs altogether, because changing the epoch is the way refineShardKey indicates that the shard key has changed.&lt;/p&gt;</comment>
                            <comment id="2549181" author="matthew.saltz" created="Tue, 19 Nov 2019 15:53:50 +0000"  >&lt;p&gt;This ticket sprouted out of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-43690?focusedCommentId=2523612&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-2523612&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;this comment&lt;/a&gt;. It&apos;s not about getting rid of epochs (though I also support that), but it&apos;s about correcting cases now where we say&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;if (epoch != currentEpoch) { &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    /* collection must have been dropped and recreated */ &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;e.g. &lt;a href=&quot;https://github.com/mongodb/mongo/blob/0bed872351365fa35107e0d9818d9066642a84a4/src/mongo/db/s/migration_destination_manager.cpp#L1156&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;. Fortunately in these cases we don&apos;t care which is before or which is after, if they&apos;re different, something is wrong.&lt;/p&gt;</comment>
                            <comment id="2548516" author="kaloian.manassiev" created="Tue, 19 Nov 2019 07:36:29 +0000"  >&lt;p&gt;What is the expected outcome of this ticket? Do you want to get rid of epochs altogether (which I support!) or is it about correctness?&lt;/p&gt;

&lt;p&gt;Also, UUIDs have the same problem like epochs with respect to drop and create, because you can&apos;t differentiate which one happened first.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="943371">SERVER-43690</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 19 Nov 2019 07:36:29 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 50 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-812</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 50 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>esha.maharishi@mongodb.com</customfieldvalue>
            <customfieldvalue>jack.mulrow@mongodb.com</customfieldvalue>
            <customfieldvalue>kaloian.manassiev@mongodb.com</customfieldvalue>
            <customfieldvalue>matthew.saltz@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hw40x3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr6qjj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3572">Sharding 2020-02-24</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hw3n6f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>