<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 06:43:05 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-80258] Failover stuck in catch up mode</title>
                <link>https://jira.mongodb.org/browse/SERVER-80258</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hi &#8211; We recently encountered a potential bug in Mongo where the failover can get stuck in catch up mode.&lt;/p&gt;

&lt;p&gt;Fundamentally &#8211; the bug seems to be sync source &lt;a href=&quot;https://github.com/mongodb/mongo/blob/v4.4/src/mongo/db/repl/topology_coordinator.cpp#L292-L420&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;selection selects a node&lt;/a&gt; with higher optime, but not necessarily the one with highest optime, whereas catch up during failover is trying to catch up to highest optime in the shard. With replication chaining disabled, this doesn&apos;t seem to work.&lt;/p&gt;

&lt;p&gt;It seems like the bug would trigger in the following scenario:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Replication Chaining is Disabled.&lt;/li&gt;
	&lt;li&gt;A shard has 5 voting nodes and 1 non voting node.&lt;/li&gt;
	&lt;li&gt;In case of unplanned failover, non-voting node is most caught up&lt;/li&gt;
	&lt;li&gt;In the failover, 2nd most caught up voting node is elected as leader.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;During the catch up mode &#8211;&#160;&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;The elected primary would determine the optime of non-voting node as the target optime.&lt;/li&gt;
	&lt;li&gt;The node would select the most caught voting node as the sync source &#8211; as we ignore non-voting nodes in the first attempt.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;This would cause the primary to get stuck until `catchupTimeoutMillis` hits.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Mongo version: 4.4&lt;/p&gt;</description>
                <environment></environment>
        <key id="2412557">SERVER-80258</key>
            <summary>Failover stuck in catch up mode</summary>
                <type id="6" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14720&amp;avatarType=issuetype">Question</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="huayu.ouyang@mongodb.com">Huayu Ouyang</assignee>
                                    <reporter username="surajn.vnit@gmail.com">Suraj Narkhede</reporter>
                        <labels>
                            <label>Bug</label>
                    </labels>
                <created>Tue, 8 Aug 2023 22:49:12 +0000</created>
                <updated>Thu, 24 Aug 2023 20:48:27 +0000</updated>
                            <resolved>Thu, 24 Aug 2023 20:48:27 +0000</resolved>
                                                                                        <votes>1</votes>
                                    <watches>12</watches>
                                                                                                                <comments>
                            <comment id="5657600" author="huayu.ouyang" created="Thu, 24 Aug 2023 19:51:30 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=surajn.vnit%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;surajn.vnit@gmail.com&quot;&gt;surajn.vnit@gmail.com&lt;/a&gt; &lt;br/&gt;
&quot;This is not true if replication chaining is disabled though, right?&quot; - yes, sorry, in the case when replication chaining is disabled, after the fix from 4.4.10, even if the primary-elect is not necessarily syncing from the most up-to-date node, and that node is syncing from the primary-elect, the primary-elect will re-evaluate its sync source after it catches up with the currently chosen one, detect the cycle, and switch sync sources to eventually choose the most up-to-date node. &lt;/p&gt;

&lt;p&gt;&quot;I was just curious about the trade-offs of doing approach one here.&quot; - I believe we decided not to implement approach one to choose the most up-to-date node as the sync source because the problem described only occurs when chaining is disabled, and approach two directly fixes the primary catchup problem for when chaining is disabled rather than changing how sync source selection works as a whole. Additionally, choosing the nearest sync source instead of the most up-to-date node is generally faster and cheaper since the most up-to-date node might be in a different region/data center, and it also aligns with how sync source selection works during steady state replication instead of having to change sync source selection specifically for primary catchup.&lt;/p&gt;</comment>
                            <comment id="5656968" author="surajn.vnit@gmail.com" created="Thu, 24 Aug 2023 16:41:08 +0000"  >&lt;p&gt;We are validating the fix in 4.4.10, I will circle back on that. Seems like it should work. Thanks for the help. Feel free to resolve the ticket.&lt;/p&gt;</comment>
                            <comment id="5656965" author="surajn.vnit@gmail.com" created="Thu, 24 Aug 2023 16:39:19 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=huayu.ouyang%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;huayu.ouyang@mongodb.com&quot;&gt;huayu.ouyang@mongodb.com&lt;/a&gt; &#8211; Thanks for the response.&#160;&lt;/p&gt;

&lt;p&gt;&quot;In general, during primary catchup, the primary-elect might not necessarily sync from the most up-to-date node, but its sync source will sync from a more up-to-date node. This will mean that the primary-elect will still be able to catchup to the most caught up secondary.&quot; &#8211;&#160; This is not true if replication chaining is disabled though, right?&lt;/p&gt;

&lt;p&gt;As per comment on the &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-58988?focusedCommentId=3973998&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-3973998&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;fix&lt;/a&gt; in 4.4.10 &#8211; &quot;we think there are two things that can be done to avoid the sync source selection cycle. First when selecting sync source when we&apos;re in primary catchup, instead of choosing based on ping latency, we can sort the candidates based on whoever is newer in oplog and select the most up-to-date node that is alive. Another thing to do is when checking &lt;tt&gt;shouldChangeSyncSource&lt;/tt&gt; after each batch, we return true if we are primary and already caught up to the sync source, but the sync source is also syncing from us, thus breaking the cycle.&quot;&#160;&lt;/p&gt;

&lt;p&gt;I was just curious about the trade-offs of doing approach one here.&#160;&lt;/p&gt;</comment>
                            <comment id="5651616" author="huayu.ouyang" created="Tue, 22 Aug 2023 19:29:18 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=surajn.vnit%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;surajn.vnit@gmail.com&quot;&gt;surajn.vnit@gmail.com&lt;/a&gt;, oplogFetcherSteadyStateMaxFetcherRestarts&#160;controls the maximum number of times the oplog fetcher will consecutively restart tailing the oplog on non-cancellation errors during steady state replication, so setting it to 0 just means if there&apos;s an error with oplog fetching during primary catchup , the primary-elect will re-do sync source selection for primary catchup. &lt;/p&gt;

&lt;p&gt;Are you looking for the catchUpTimeoutMillis field? Setting this to 0 will make it so that newly elected primaries will not attempt to catch up, so it will become primary immediately before it is fully caught up. However, in this case, the primary would not roll back, and instead the non-voting node that is ahead of the primary would rollback its oplog entries that are ahead of the primary (and these oplog entries wouldn&apos;t have been majority committed by 3/5 of the voting nodes because if they were, the newly elected primary that is 2nd most caught up would not have been able to been elected by 3/5 voting nodes).&lt;/p&gt;

&lt;p&gt;I&apos;m not sure what you mean by &quot;What are the trade-offs of selecting the most caught up secondary as sync source directly during _catchup phase?&quot;, could you elaborate? Thanks! In general, during primary catchup, the primary-elect might not necessarily sync from the most up-to-date node, but its sync source will sync from a more up-to-date node. This will mean that the primary-elect will still be able to catchup to the most caught up secondary.&lt;/p&gt;</comment>
                            <comment id="5648634" author="surajn.vnit@gmail.com" created="Mon, 21 Aug 2023 22:37:53 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=huayu.ouyang%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;huayu.ouyang@mongodb.com&quot;&gt;huayu.ouyang@mongodb.com&lt;/a&gt; &#8211; Thanks for digging into this.&lt;/p&gt;

&lt;p&gt;I did see the cycle as mentioned in the &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-58988&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;bug&lt;/a&gt; during the incident. So, this fix would have likely helped with the faster failover for us.&lt;/p&gt;

&lt;p&gt;I have a follow up question &#8211; we have &lt;em&gt;oplogFetcherSteadyStateMaxFetcherRestarts&lt;/em&gt;&#160;_&lt;em&gt;&#160;= 0. Will this cause new primary to assume primary before its fully caught up and rollback on the node that was most ahead in oplog? What are the trade-offs of selecting the most caught up secondary as sync source directly during _catchup&lt;/em&gt; phase?&lt;/p&gt;</comment>
                            <comment id="5647639" author="huayu.ouyang" created="Mon, 21 Aug 2023 17:30:11 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=surajn.vnit%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;surajn.vnit@gmail.com&quot;&gt;surajn.vnit@gmail.com&lt;/a&gt;, I can&apos;t confirm if this is definitely the cause without seeing the logs, but an issue with sync source selection cycles during primary catchup was fixed in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-58988&quot; title=&quot;Avoid sync source selection cycle during primary catchup.&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-58988&quot;&gt;&lt;del&gt;SERVER-58988&lt;/del&gt;&lt;/a&gt;, which was released in 4.4.10, so it&apos;s possible that upgrading to &amp;gt;=4.4.10 would fix your case.&lt;/p&gt;</comment>
                            <comment id="5647410" author="surajn.vnit@gmail.com" created="Mon, 21 Aug 2023 16:34:11 +0000"  >&lt;p&gt;It&apos;s 4.4.9.&#160;&lt;/p&gt;</comment>
                            <comment id="5646697" author="huayu.ouyang" created="Mon, 21 Aug 2023 13:49:59 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=surajn.vnit%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;surajn.vnit@gmail.com&quot;&gt;surajn.vnit@gmail.com&lt;/a&gt;, do you also know which version of mongoDB this was on (like 4.4.0 vs 4.4.1, etc)?&lt;/p&gt;</comment>
                            <comment id="5644314" author="huayu.ouyang" created="Fri, 18 Aug 2023 20:52:15 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=surajn.vnit%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;surajn.vnit@gmail.com&quot;&gt;surajn.vnit@gmail.com&lt;/a&gt; Just to confirm, we don&apos;t have any logs from this incident either?&lt;/p&gt;</comment>
                            <comment id="5638214" author="surajn.vnit@gmail.com" created="Wed, 16 Aug 2023 16:33:40 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=chris.kelly%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;chris.kelly@mongodb.com&quot;&gt;chris.kelly@mongodb.com&lt;/a&gt; . I actually linked internal code link (not accessible) in the original description. Can you please update the link with &lt;a href=&quot;https://github.com/mongodb/mongo/blob/v4.4/src/mongo/db/repl/topology_coordinator.cpp#L292-L420.&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/v4.4/src/mongo/db/repl/topology_coordinator.cpp#L292-L420.&lt;/a&gt; Thanks!&lt;/p&gt;</comment>
                            <comment id="5633264" author="JIRAUSER1265262" created="Mon, 14 Aug 2023 19:10:11 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=surajn.vnit%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;surajn.vnit@gmail.com&quot;&gt;surajn.vnit@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;I&apos;ve passed this on to the relevant team to look into your scenario further, in case we can say something definitive based off of it. If you happen to get logs again, please upload them (as it may prove helpful to look into this behavior further).&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="5633094" author="surajn.vnit@gmail.com" created="Mon, 14 Aug 2023 18:17:56 +0000"  >&lt;p&gt;Hello &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=chris.kelly%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;chris.kelly@mongodb.com&quot;&gt;chris.kelly@mongodb.com&lt;/a&gt; &#8211; thanks for your help! Anything I can help with to investigate this issue?&lt;/p&gt;</comment>
                            <comment id="5627578" author="surajn.vnit@gmail.com" created="Thu, 10 Aug 2023 23:15:37 +0000"  >&lt;p&gt;Hello &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=chris.kelly%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;chris.kelly@mongodb.com&quot;&gt;chris.kelly@mongodb.com&lt;/a&gt; &#8211; thanks for checking this. Unfortunately, we do not have the diagnostic data going back to 08/05 on shard now.&lt;/p&gt;

&lt;p&gt;I can shed more light here to explain the scenario:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Let&apos;s say &#8211; shard has 6 Nodes (A, B, C, D, E, F) with A being the primary.&lt;/li&gt;
	&lt;li&gt;A, B, C, D, E are hidden=false, priority=1, vote=1&lt;/li&gt;
	&lt;li&gt;F is hidden=true, priority=0, vote=0&lt;/li&gt;
	&lt;li&gt;At t1, node A crashed.&#160;&lt;/li&gt;
	&lt;li&gt;After A crashed, nodes in the decreasing order of optimes are F, B, C, D, E.&lt;/li&gt;
	&lt;li&gt;Node C triggered the election and it got two votes (from D and E) and it elected itself as primary.
	&lt;ul&gt;
		&lt;li&gt;Mongo logs: &quot;Election succeeded, assuming primary role&quot;&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
	&lt;li&gt;Node C, via hearts determine the target optime as optime of F.&#160;
	&lt;ul&gt;
		&lt;li&gt;&apos;{&quot;t&quot;:
{&quot;$date&quot;:&quot;2023-08-05T19:41:58.583+00:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;I&quot;, &quot;c&quot;:&quot;REPL&quot;, &quot;id&quot;:21365, &quot;ctx&quot;:&quot;ReplCoord-1525&quot;,&quot;msg&quot;:&quot;Heartbeats updated catchup target optime&quot;,&quot;attr&quot;:{&quot;targetOpTime&quot;:{&quot;ts&quot;:{&quot;$timestamp&quot;:{&quot;t&quot;:1691264511,&quot;i&quot;:983}},&quot;t&quot;:102}}}&apos;&lt;/p&gt;&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
	&lt;li&gt;But Node C, chose Node B as sync source because chose sync source just looks for a node with higher optime than the chosen primary.&lt;/li&gt;
	&lt;li&gt;Now, C is trying to catchup with optime of F but B is behind F and replication chaining is disabled. So the failover is stuck in catch up mode.&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="5627104" author="JIRAUSER1265262" created="Thu, 10 Aug 2023 19:18:58 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=surajn.vnit%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;surajn.vnit@gmail.com&quot;&gt;surajn.vnit@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thanks for your report. To investigate this further, it&apos;d be helpful to get some diagnostic data covering the time you&apos;re observing this happen.&lt;/p&gt;

&lt;p&gt;I&apos;ve created a secure &lt;a href=&quot;https://amphora.corp.mongodb.com/public/upload/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJmb2xkZXJfaWQiOiIyMjEzMTE0MjczNDgiLCJleHAiOjE2OTQyODcxMDl9.h0CMjQo83hpB5PylVgxmPfBxYui-uKm9V7Rsf2gjAtI&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;upload link&lt;/a&gt;&#160;for you. Files uploaded to this portal are hosted on Box, are visible only to MongoDB employees, and are routinely deleted after some time.&lt;/p&gt;

&lt;p&gt;For each node in the replica set spanning a time period that includes the incident, would you please archive (tar or zip) and upload to that link:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;the mongod logs&lt;/li&gt;
	&lt;li&gt;the &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt; directory (the contents are described &lt;a href=&quot;https://docs.mongodb.com/manual/administration/analyzing-mongodb-performance/#full-time-diagnostic-data-capture&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;)&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Chris&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>14.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 10 Aug 2023 19:18:58 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        23 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>huayu.ouyang@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            23 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>chris.kelly@mongodb.com</customfieldvalue>
            <customfieldvalue>huayu.ouyang@mongodb.com</customfieldvalue>
            <customfieldvalue>surajn.vnit@gmail.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i2lgrz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i23jh4:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="7627">Repl 2023-09-04</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[chris.kelly@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i2l2xb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>