<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:55:08 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-40487] Stop running the RstlKillOpthread when a node is no longer primary</title>
                <link>https://jira.mongodb.org/browse/SERVER-40487</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Currently, when 2 concurrent step downs are triggered (can be a combination of &lt;a href=&quot;https://github.com/mongodb/mongo/blob/3b2fb3741dc4df0a6578ba6485584bf6cc3fb586/src/mongo/db/repl/replication_coordinator_impl.cpp#L1896&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;conditional step down&lt;/a&gt; and &lt;a href=&quot;https://github.com/mongodb/mongo/blob/3b2fb3741dc4df0a6578ba6485584bf6cc3fb586/src/mongo/db/repl/replication_coordinator_impl_heartbeat.cpp#L286-L288&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;unconditional step down&lt;/a&gt; or 2 conditional step downs), there is a possibility that the step down thread can kill the transaction operations processed by the second oplog application.&lt;/p&gt;

&lt;p&gt;Consider the below scenario and assume that node A is in primary state.&lt;br/&gt;
 1) User executes replSetStepDown cmd (&lt;font color=&quot;#403294&quot;&gt;&lt;b&gt;Thread X&lt;/b&gt;&lt;/font&gt;).&lt;br/&gt;
 2) Thread X is at &lt;a href=&quot;https://github.com/mongodb/mongo/blob/3b2fb3741dc4df0a6578ba6485584bf6cc3fb586/src/mongo/db/repl/replication_coordinator_impl.cpp#L1909&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this&#160;&lt;/a&gt;line.&lt;br/&gt;
 3) Now, node A notices that a new term has begun via heartbeat. So, node A&#160;&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/3b2fb3741dc4df0a6578ba6485584bf6cc3fb586/src/mongo/db/repl/replication_coordinator_impl_heartbeat.cpp#L380&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;steps down&lt;/a&gt; via unconditional stepdown code path.&lt;br/&gt;
 4) Now the state of node A will be SECONDARY.&lt;br/&gt;
 5) Node A&apos;s oplog application tries to apply the prepare/commit oplog entry. This would require the secondary oplog application to checkout the session. Let assume, oplog application &lt;b&gt;&lt;font color=&quot;#00875a&quot;&gt;thread Y&lt;/font&gt;,&lt;/b&gt; tries to apply commit oplog entry and is at &lt;a href=&quot;https://github.com/mongodb/mongo/blob/3b2fb3741dc4df0a6578ba6485584bf6cc3fb586/src/mongo/db/repl/transaction_oplog_application.cpp#L180&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this&lt;/a&gt; line.&lt;br/&gt;
 6) Read operations comes in (&lt;font color=&quot;#de350b&quot;&gt;&lt;b&gt;Thread Z&lt;/b&gt;&lt;/font&gt;), acquired the RSTL lock in mode IX&#160; and global lock in IS mode. And, its &lt;b&gt;blocked by&#160; thread Y due to prepare conflict ( conflict at the document lock)&lt;/b&gt;.&lt;br/&gt;
 7) Thread X resumes and &lt;a href=&quot;https://github.com/mongodb/mongo/blob/3b2fb3741dc4df0a6578ba6485584bf6cc3fb586/src/mongo/db/repl/replication_coordinator_impl.cpp#L1910-L1911&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;enqueues the RSTL lock in X mode&lt;/a&gt; as it is &lt;b&gt;blocked by read thread (thread Z)&lt;/b&gt;.&lt;br/&gt;
 8) Thread X starts &quot;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/3b2fb3741dc4df0a6578ba6485584bf6cc3fb586/src/mongo/db/repl/replication_coordinator_impl.cpp#L1837&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;RstlKillOpthread&lt;/a&gt;&quot;. Now, RstlKillOpthread &lt;b&gt;marks the thread Y(belongs to secondary oplog application) as killed as part of &lt;a href=&quot;https://github.com/mongodb/mongo/blob/3b2fb3741dc4df0a6578ba6485584bf6cc3fb586/src/mongo/db/repl/replication_coordinator_impl.cpp#L1854-L1855&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;killSessionsAbortUnpreparedTransactions&lt;/a&gt;&lt;/b&gt;.&lt;/p&gt;</description>
                <environment></environment>
        <key id="730211">SERVER-40487</key>
            <summary>Stop running the RstlKillOpthread when a node is no longer primary</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="10038" iconUrl="https://jira.mongodb.org/images/icons/subtask.gif" description="">Backlog</status>
                    <statusCategory id="2" key="new" colorName="default"/>
                                    <resolution id="-1">Unresolved</resolution>
                                        <assignee username="backlog-server-repl">Backlog - Replication Team</assignee>
                                    <reporter username="suganthi.mani@mongodb.com">Suganthi Mani</reporter>
                        <labels>
                    </labels>
                <created>Thu, 4 Apr 2019 22:25:46 +0000</created>
                <updated>Tue, 6 Dec 2022 03:02:20 +0000</updated>
                                                                            <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="2284031" author="judah.schvimer" created="Thu, 13 Jun 2019 20:36:24 +0000"  >&lt;p&gt;We will re-evaluate this after PM-1455. Jason Carey has a plan to improve how operations choose to run only as a primary. We should sync up with him before starting this work.&lt;/p&gt;</comment>
                            <comment id="2257068" author="judah.schvimer" created="Wed, 22 May 2019 21:03:51 +0000"  >&lt;p&gt;This ticket will be for the optimization to kill fewer readers.&lt;/p&gt;</comment>
                            <comment id="2254887" author="judah.schvimer" created="Tue, 21 May 2019 19:37:44 +0000"  >&lt;blockquote&gt;
&lt;p&gt;We will also double check in the deadlock fix design that there were no other deadlocks that we were concerned about.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;I have done this check and don&apos;t see anything further to do besides commit the test Suganthi wrote.&lt;/p&gt;</comment>
                            <comment id="2253037" author="judah.schvimer" created="Mon, 20 May 2019 17:48:33 +0000"  >&lt;blockquote&gt;
&lt;p&gt;In 4.0, we won&apos;t get into this problem as we don&apos;t support cross-shard transaction.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;We don&apos;t hit some of the problem, but not necessarily all of it. The concern with killing secondary oplog application is still there in 4.0 since it was not transaction specific. I think we&apos;re safe because both in 4.2 and 4.0 we only kill user operations and secondary oplog application is not a user operation so it won&apos;t be killed.&lt;/p&gt;</comment>
                            <comment id="2249421" author="judah.schvimer" created="Thu, 16 May 2019 19:37:18 +0000"  >&lt;blockquote&gt;
&lt;p&gt;We have to check if a secondary could see kInProgress for a transaction here.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;It is possible, however when we try to abort the transaction we will &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a48102efbc35b8c622eb72599c5eb3fa3c47c430/src/mongo/db/kill_sessions_local.cpp#L72&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;check out the session&lt;/a&gt;. We invariant &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a48102efbc35b8c622eb72599c5eb3fa3c47c430/src/mongo/db/session_catalog_mongod.cpp#L417-L418&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt; that the transaction is in a &quot;prepared&quot; state before it is checked back in during secondary oplog application. When we try to abort a prepared transaction during stepdown we will &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a48102efbc35b8c622eb72599c5eb3fa3c47c430/src/mongo/db/transaction_participant.cpp#L1377-L1385&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;ignore the attempt&lt;/a&gt; and leave the transaction in prepare. This was all investigated thoroughly in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-37348&quot; title=&quot;TransactionReaper and periodic transaction abort thread shouldn&amp;#39;t abort transactions on secondaries&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-37348&quot;&gt;&lt;del&gt;SERVER-37348&lt;/del&gt;&lt;/a&gt; for the TransactionReaper and abort threads.&lt;/p&gt;</comment>
                            <comment id="2249369" author="judah.schvimer" created="Thu, 16 May 2019 19:02:28 +0000"  >&lt;p&gt;We have to check if a secondary could see &lt;tt&gt;kInProgress&lt;/tt&gt; for a transaction &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a48102efbc35b8c622eb72599c5eb3fa3c47c430/src/mongo/db/kill_sessions_local.cpp#L65-L69&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;

&lt;p&gt;We do check if we&apos;re secondary before we yield locks for prepared transactions.&lt;/p&gt;

&lt;p&gt;We will also double check in the deadlock fix design that there were no other deadlocks that we were concerned about.&lt;/p&gt;</comment>
                            <comment id="2249163" author="suganthi.mani" created="Thu, 16 May 2019 17:27:57 +0000"  >&lt;p&gt;Spoke to &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=judah.schvimer&quot; class=&quot;user-hover&quot; rel=&quot;judah.schvimer&quot;&gt;judah.schvimer&lt;/a&gt;, we won&apos;t be doing the optimization now. But, we would be adding a jstest for this scenario.&lt;/p&gt;</comment>
                            <comment id="2248204" author="suganthi.mani" created="Wed, 15 May 2019 22:41:12 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=judah.schvimer&quot; class=&quot;user-hover&quot; rel=&quot;judah.schvimer&quot;&gt;judah.schvimer&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Currently a step down thread can wait for RSTL lock when the node has stepped down and transitioned to secondary state. Consider the below concurrent step down code paths.&lt;/p&gt;

&lt;p&gt;1) &lt;font color=&quot;#00875a&quot;&gt;Unconditional step down&lt;/font&gt; &amp;amp;&#160;&lt;font color=&quot;#de350b&quot;&gt;Conditional step down (step down cmd)&lt;/font&gt;.&lt;/p&gt;

&lt;p&gt;2) &lt;font color=&quot;#00875a&quot;&gt;Force reconfig&lt;/font&gt;&#160; &amp;amp;&lt;font color=&quot;#de350b&quot;&gt; step down via heartbeat.&lt;/font&gt;&lt;/p&gt;

&lt;p&gt;3) &lt;font color=&quot;#00875a&quot;&gt;step down via heartbeat&lt;/font&gt; &amp;amp;&#160;&#160;&lt;font color=&quot;#de350b&quot;&gt;Force reconfig&lt;/font&gt;.&lt;/p&gt;

&lt;p&gt;In above 3 cases, the former(&lt;font color=&quot;#00875a&quot;&gt;green&lt;/font&gt;) wins and pushing the later ones(&lt;font color=&quot;#de350b&quot;&gt;red&lt;/font&gt;) to enqueue the RSTL, starting the killop thread and waiting for RSTL lock when the node is in secondary state.&lt;/p&gt;

&lt;p&gt;As per the new approach (&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40700&quot; title=&quot;Deadlock between read prepare conflicts and state transitions&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40700&quot;&gt;&lt;del&gt;SERVER-40700&lt;/del&gt;&lt;/a&gt;), even though our step downs take RSTL lock in X mode, we kill read operations blocked on prepared txn (secondary oplog application). This means we won&apos;t get 3 way deadlock (problem No:2).&lt;/p&gt;

&lt;p&gt;Next question, is do we need to do optimization of not killing the read operations when they are no longer primary. Currently, I have 2 solutions and both of them have some flaws/complications.&lt;/p&gt;

&lt;p&gt;1) (WIP)&#160;Listener&#160;model - where step down registers the listener&#160;and the winner&#160; should interrupt all the listeners. When they interrupt the listeners, they have to kill the listener&apos;s killopthread and mark the listener thread as killed which means this solution has 2 problems&lt;/p&gt;

&lt;p&gt;&#160; &#160; &#160; - It can kill the stepdown via hb (case 2 - internal operation) and reconfig cmd (case 3) . And, that&apos;s not correct.&lt;/p&gt;

&lt;p&gt;&#160; &#160; &#160; -&#160; For case 1,&#160; since we can take RSTL lock in an uninterruptible lock guard, we can&apos;t make the waiting for lock acquisition to be interrupted and there is a possibility of 3 way deadlock.&lt;/p&gt;

&lt;p&gt;2) After every iteration, check if the member state is primary in the killOpthread. If not, break the loop (ie) stop the killop thread. Now comes the complication of stopping the step down thread from waiting for RSTL lock as&#160; ReplicationStateTransitionLockGuard is not&#160; thread safe. So, the fix is not going to be straightforward.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;With the complications, is it worth implementing the optimization&lt;/b&gt;?.&#160; To be noted, we kill the read operations with the retryable&#160;error code ErrorCodes.InterruptedDueToStepDown. Since this killing read operations on secondary is only for a brief window,&#160;I feel its ok to close this ticket or decrease the priority of this ticket.&lt;/p&gt;

&lt;p&gt;Note: In 4.0, we won&apos;t get into this problem as we don&apos;t support cross-shard transaction.&lt;/p&gt;</comment>
                            <comment id="2237111" author="suganthi.mani" created="Tue, 7 May 2019 15:19:34 +0000"  >&lt;p&gt;Step down thread is no longer going to take RSTL in S mode but in X mode. So, the problem described in this ticket is still valid.&lt;/p&gt;</comment>
                            <comment id="2223916" author="suganthi.mani" created="Wed, 24 Apr 2019 19:57:34 +0000"  >&lt;p&gt;After, &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40700&quot; title=&quot;Deadlock between read prepare conflicts and state transitions&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40700&quot;&gt;&lt;del&gt;SERVER-40700&lt;/del&gt;&lt;/a&gt;, problem No:2 (3-way deadlock) is also not possible, as the step down thread is going to take the lock in S mode which should not conflict with the read thread. So, there is nothing to do for this ticket. So, marking this ticket as depends on&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40700&quot; title=&quot;Deadlock between read prepare conflicts and state transitions&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40700&quot;&gt;&lt;del&gt;SERVER-40700&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="2223915" author="suganthi.mani" created="Wed, 24 Apr 2019 19:57:18 +0000"  >&lt;p&gt;Currently, we see 2 problems over here.&lt;br/&gt;
 Problem No:1 Operations applied during secondary oplog application can be marked killed by step down.&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Step No: 8 is not possible because the operations run via secondary oplog application are not interruptible&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/3b2fb3741dc4df0a6578ba6485584bf6cc3fb586/src/mongo/db/repl/sync_tail.cpp#L1277&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;(OperationContext::runWithoutInterruptionExceptAtGlobalShutdown)&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Problem No:2 Now, if the secondary oplog application aren&apos;t killed. Then it can lead to 3-way deadlock.&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Read thread (&lt;font color=&quot;#de350b&quot;&gt;&lt;b&gt;thread Z&lt;/b&gt;&lt;/font&gt;) blocked by prepared transaction &lt;b&gt;&lt;font color=&quot;#00875a&quot;&gt;thread Y&lt;/font&gt;&lt;/b&gt; (secondary oplog application).&lt;/li&gt;
	&lt;li&gt;Step down (&lt;b&gt;&lt;font color=&quot;#403294&quot;&gt;thread X&lt;/font&gt;&lt;/b&gt;) blocked by read thread.&lt;/li&gt;
	&lt;li&gt;prepared transaction &lt;a href=&quot;https://github.com/mongodb/mongo/blob/3b2fb3741dc4df0a6578ba6485584bf6cc3fb586/src/mongo/db/transaction_participant.cpp#L1199&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;blocked&lt;/a&gt; by step down thread.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;To solve it, we need to implement a way such that the stepdown thread stops running the RstlKillOpthread and waiting for RSTL lock when the node is no longer in PRIMARY state.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="617120">SERVER-37574</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10620">
                    <name>Issue split</name>
                                            <outwardlinks description="split to">
                                        <issuelink>
            <issuekey id="774862">SERVER-41283</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="740577">SERVER-40700</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="610864">SERVER-37348</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>11.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25128"><![CDATA[Replication]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 9 May 2019 18:11:17 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 34 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/SERVER-37574'>SERVER-37574</a></s>, <s><a href='https://jira.mongodb.org/browse/PM-1455'>PM-1455</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>alexander.golin@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 34 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-server-repl</customfieldvalue>
            <customfieldvalue>judah.schvimer@mongodb.com</customfieldvalue>
            <customfieldvalue>suganthi.mani@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hut3k7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|huimjz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2918">Repl 2019-04-22</customfieldvalue>
    <customfieldvalue id="2919">Repl 2019-05-06</customfieldvalue>
    <customfieldvalue id="2920">Repl 2019-05-20</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|husptj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>