<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:46:42 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-59226] Deadlock when stepping down with a profile session marked as uninterruptible</title>
                <link>https://jira.mongodb.org/browse/SERVER-59226</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;When a primary is stepping down, when calling &lt;tt&gt;_stepDownFinish&lt;/tt&gt; it kills all sessions through &lt;tt&gt;invalidateSessionsForStepDown -&amp;gt; killSessionsAction -&amp;gt; checkOutSessionForKill&lt;/tt&gt;. &lt;a href=&quot;https://github.com/mongodb/mongo/blob/f3e8bfb0ea52ae167e097f3f3fd9bf183e6b4a8a/src/mongo/db/repl/replication_coordinator_impl_heartbeat.cpp#L532-L548&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;This is after it has grabbed the RSTL via &lt;tt&gt;AutoGetRstlForStepUpStepDown&lt;/tt&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;In the process of stepping down, if it is trying to kill an already checked out session, there is a potential for deadlock, as it needs to wait until the session is checked back in.&lt;/p&gt;

&lt;p&gt;The session will end up getting interrupted when it tries to grab a lock such as the &lt;tt&gt;GlobalLock&lt;/tt&gt;. However, if the session&apos;s opCtx is marked as uninterruptible, then it is possible that the checked out session is waiting on the &lt;tt&gt;GlobalLock&lt;/tt&gt;, while the step down thread (which has the RSTL) is waiting on the checked out session, causing a deadlock.&lt;/p&gt;

&lt;p&gt;This is possible when &lt;a href=&quot;https://github.com/mongodb/mongo/blob/f3e8bfb0ea52ae167e097f3f3fd9bf183e6b4a8a/src/mongo/db/introspect.cpp#L107-L117&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;profiling&lt;/a&gt;. In general, it may be possible with other uses of the &lt;tt&gt;UninterruptibleLockGuard&lt;/tt&gt;.&lt;/p&gt;
</description>
                <environment></environment>
        <key id="1847496">SERVER-59226</key>
            <summary>Deadlock when stepping down with a profile session marked as uninterruptible</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="wenbin.zhu@mongodb.com">Wenbin Zhu</assignee>
                                    <reporter username="vishnu.kaushik@mongodb.com">Vishnu Kaushik</reporter>
                        <labels>
                    </labels>
                <created>Tue, 10 Aug 2021 17:24:18 +0000</created>
                <updated>Sun, 29 Oct 2023 21:49:42 +0000</updated>
                            <resolved>Sat, 4 Sep 2021 01:51:41 +0000</resolved>
                                                    <fixVersion>4.4.11</fixVersion>
                    <fixVersion>4.2.18</fixVersion>
                    <fixVersion>5.0.4</fixVersion>
                    <fixVersion>5.1.0-rc0</fixVersion>
                                                        <votes>0</votes>
                                    <watches>31</watches>
                                                                                                                <comments>
                            <comment id="4161765" author="xgen-internal-githook" created="Mon, 1 Nov 2021 20:51:48 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Wenbin Zhu&apos;, &apos;email&apos;: &apos;wenbin.zhu@mongodb.com&apos;, &apos;username&apos;: &apos;WenbinZhu&apos;}
&lt;p&gt;Message: Revert &quot;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-59226&quot; title=&quot;Deadlock when stepping down with a profile session marked as uninterruptible&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-59226&quot;&gt;&lt;del&gt;SERVER-59226&lt;/del&gt;&lt;/a&gt; Fix deadlock between uninterruptible profiling operation and stepdown thread.&quot;&lt;/p&gt;

&lt;p&gt;This reverts commit d1a9e68d581f8e3162bc790bdb30986b8d030d86.&lt;br/&gt;
Branch: v4.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/8443b63b12e53d21590b1f7dcf02c5df26b9ff40&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/8443b63b12e53d21590b1f7dcf02c5df26b9ff40&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4161709" author="xgen-internal-githook" created="Mon, 1 Nov 2021 20:27:38 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Wenbin Zhu&apos;, &apos;email&apos;: &apos;wenbin.zhu@mongodb.com&apos;, &apos;username&apos;: &apos;WenbinZhu&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-59226&quot; title=&quot;Deadlock when stepping down with a profile session marked as uninterruptible&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-59226&quot;&gt;&lt;del&gt;SERVER-59226&lt;/del&gt;&lt;/a&gt; Fix deadlock between uninterruptible profiling operation and stepdown thread.&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 43479818bd01f27ee25b6e992045529d2ac0185a)&lt;br/&gt;
Branch: v4.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/d1a9e68d581f8e3162bc790bdb30986b8d030d86&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/d1a9e68d581f8e3162bc790bdb30986b8d030d86&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4139247" author="xgen-internal-githook" created="Thu, 21 Oct 2021 15:56:13 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Wenbin Zhu&apos;, &apos;email&apos;: &apos;wenbin.zhu@mongodb.com&apos;, &apos;username&apos;: &apos;WenbinZhu&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-59226&quot; title=&quot;Deadlock when stepping down with a profile session marked as uninterruptible&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-59226&quot;&gt;&lt;del&gt;SERVER-59226&lt;/del&gt;&lt;/a&gt; Fix deadlock between uninterruptible profiling operation and stepdown thread.&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 43479818bd01f27ee25b6e992045529d2ac0185a)&lt;br/&gt;
Branch: v4.2&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/46d180e4f463f96b7d5f64c73e43204fd47f5fd4&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/46d180e4f463f96b7d5f64c73e43204fd47f5fd4&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4136920" author="xgen-internal-githook" created="Wed, 20 Oct 2021 23:45:37 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Wenbin Zhu&apos;, &apos;email&apos;: &apos;wenbin.zhu@mongodb.com&apos;, &apos;username&apos;: &apos;WenbinZhu&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-59226&quot; title=&quot;Deadlock when stepping down with a profile session marked as uninterruptible&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-59226&quot;&gt;&lt;del&gt;SERVER-59226&lt;/del&gt;&lt;/a&gt; Fix deadlock between uninterruptible profiling operation and stepdown thread.&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 43479818bd01f27ee25b6e992045529d2ac0185a)&lt;br/&gt;
Branch: v4.4&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/73ab1b397b1292dd791775e9f2f73fb0b3b4699b&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/73ab1b397b1292dd791775e9f2f73fb0b3b4699b&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4124421" author="xgen-internal-githook" created="Thu, 14 Oct 2021 16:24:35 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Wenbin Zhu&apos;, &apos;email&apos;: &apos;wenbin.zhu@mongodb.com&apos;, &apos;username&apos;: &apos;WenbinZhu&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-59226&quot; title=&quot;Deadlock when stepping down with a profile session marked as uninterruptible&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-59226&quot;&gt;&lt;del&gt;SERVER-59226&lt;/del&gt;&lt;/a&gt; Fix deadlock between uninterruptible profiling operation and stepdown thread.&lt;br/&gt;
Branch: v5.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/ed83e49f6388e1c1260b610aff0c54fdd66389c9&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/ed83e49f6388e1c1260b610aff0c54fdd66389c9&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4040177" author="xgen-internal-githook" created="Sat, 4 Sep 2021 00:03:54 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Wenbin Zhu&apos;, &apos;email&apos;: &apos;wenbin.zhu@mongodb.com&apos;, &apos;username&apos;: &apos;WenbinZhu&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-59226&quot; title=&quot;Deadlock when stepping down with a profile session marked as uninterruptible&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-59226&quot;&gt;&lt;del&gt;SERVER-59226&lt;/del&gt;&lt;/a&gt; Fix deadlock between uninterruptible profiling operation and stepdown thread.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/43479818bd01f27ee25b6e992045529d2ac0185a&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/43479818bd01f27ee25b6e992045529d2ac0185a&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4029899" author="JIRAUSER1258790" created="Mon, 30 Aug 2021 18:59:43 +0000"  >&lt;p&gt;After discussing with execution team, we decided to push the fix for now, and later investigate the ideal solutions, such as removing RSTL for non-replicated write and profiling in a separate thread/executor. I have created&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-59673&quot; title=&quot;Investigate better solutions for fixing the deadlock issue in profiling&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-59673&quot;&gt;&lt;del&gt;SERVER-59673&lt;/del&gt;&lt;/a&gt; to track the investigation.&lt;/p&gt;</comment>
                            <comment id="4024855" author="schwerin" created="Thu, 26 Aug 2021 18:03:29 +0000"  >&lt;p&gt;&lt;tt&gt;OperationContext&lt;/tt&gt; shouldn&apos;t have replication-specific member fields. It&apos;s a layer violation, as can be seen by the fact that there are no replication concepts in mongos.&lt;/p&gt;</comment>
                            <comment id="4024764" author="lingzhi.deng" created="Thu, 26 Aug 2021 17:30:11 +0000"  >&lt;p&gt;I think we used &lt;tt&gt;UninterruptibleLockGuard&lt;/tt&gt; because we want to output the profiler entry to &lt;tt&gt;system.profile&lt;/tt&gt; even if the &lt;tt&gt;opCtx&lt;/tt&gt; is interrupted (e.g. the operation times out). &lt;tt&gt;system.profile&lt;/tt&gt; is a non-replicate collection. And ideally, we don&apos;t actually need to acquire RSTL for non-replicate writes and writes on &lt;tt&gt;system.profile&lt;/tt&gt; shouldn&apos;t get in the way of repl state transitions. But it might be difficult to skip RSTL in the write paths today. So &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=wenbin.zhu&quot; class=&quot;user-hover&quot; rel=&quot;wenbin.zhu&quot;&gt;wenbin.zhu&lt;/a&gt;&apos;s patch instead is to remove the &lt;tt&gt;UninterruptibleLockGuard&lt;/tt&gt; but add a flag to &lt;tt&gt;opCtx&lt;/tt&gt; to ignore interruptions except &lt;tt&gt;InterruptedDueToReplStateChange&lt;/tt&gt;.&lt;/p&gt;</comment>
                            <comment id="4023778" author="schwerin" created="Thu, 26 Aug 2021 12:10:40 +0000"  >&lt;p&gt;Why is the profiling code uninterruptible in the first place?&lt;/p&gt;</comment>
                            <comment id="4007171" author="JIRAUSER1258790" created="Wed, 18 Aug 2021 17:56:57 +0000"  >&lt;p&gt;And to answer&#160;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=josef.ahmad&quot; class=&quot;user-hover&quot; rel=&quot;josef.ahmad&quot;&gt;josef.ahmad&lt;/a&gt;&apos;s question,&#160;I believe 4.0 is also impacted, we should be&#160; able to have a 4.0 backport.&lt;/p&gt;</comment>
                            <comment id="4005579" author="JIRAUSER1258790" created="Tue, 17 Aug 2021 23:53:40 +0000"  >&lt;p&gt;I have successfully reproduced this issue in test. This only happens to retryable writes as mentioned in the previous comment. But note that only retryable updates and deletes are affected, not for inserts. This is because for inserts, the profile operation is called &lt;a href=&quot;https://github.com/mongodb/mongo/blob/70e9a9e4551034a0ee3e42b7fa66121cb494bbc8/src/mongo/db/service_entry_point_common.cpp#L2082&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;in `service_entry_point_common.cpp`&lt;/a&gt;, while for updates and deletes, the profile operation is called &lt;a href=&quot;https://github.com/mongodb/mongo/blob/70e9a9e4551034a0ee3e42b7fa66121cb494bbc8/src/mongo/db/ops/write_ops_exec.cpp#L156&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;in `write_ops_exec.cpp`&lt;/a&gt;.&#160; The difference is that for inserts, when profile() is called, the session has already been &lt;a href=&quot;https://github.com/mongodb/mongo/blob/70e9a9e4551034a0ee3e42b7fa66121cb494bbc8/src/mongo/db/session_catalog.cpp#L277-L289&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;checked in&lt;/a&gt;, so it will not block the stepdown thread on checking out session even though the opCtx is marked uninterruptible, while for updates and deletes, the session is still in checked out state when profile() is being called, which would cause deadlocks with stepdown&#160; thread.&#160;&lt;/p&gt;

&lt;p&gt;I&apos;m wondering if updates and deletes can move the profile functionality after checking back in session, like insert does, as an optimization, not related to this ticket though.&lt;/p&gt;</comment>
                            <comment id="4002654" author="JIRAUSER1258790" created="Mon, 16 Aug 2021 20:47:52 +0000"  >&lt;p&gt;This seems to only happen with retryable writes, not with transactions, because in transactions, the opCtx&apos;s locker would have the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/f3e8bfb0ea52ae167e097f3f3fd9bf183e6b4a8a/src/mongo/db/concurrency/locker.h#L132&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;maxLockTimeout&lt;/a&gt; set (the `maxTransactionLockRequestTimeoutMillis` server parameter and by default 5ms). And when profiling, if the maxLockTimeout is set, we &lt;a href=&quot;https://github.com/mongodb/mongo/blob/f3e8bfb0ea52ae167e097f3f3fd9bf183e6b4a8a/src/mongo/db/introspect.cpp#L112-L114&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;don&apos;t make the opCtx uninterruptible&lt;/a&gt;. Will try to reproduce the issue. The proposed fix could be making the opCtx interruptible when maxLockTimeout is not set and retry if the operation is interrupted until success, but don&apos;t retry when&#160;maxLockTimeout is set to keep the same behavior as today.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                                <inwardlinks description="backports">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10520">
                    <name>Problem/Incident</name>
                                            <outwardlinks description="causes">
                                        <issuelink>
            <issuekey id="1974365">SERVER-63143</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="1881402">SERVER-60161</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1863386">SERVER-59673</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="1786976">SERVER-57756</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>13.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="21777"><![CDATA[v5.0]]></customfieldvalue>
    <customfieldvalue key="18953"><![CDATA[v4.4]]></customfieldvalue>
    <customfieldvalue key="16775"><![CDATA[v4.2]]></customfieldvalue>
    <customfieldvalue key="15640"><![CDATA[v4.0]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[5002K00000s1RJYQA2, 5002K00000wYxSNQA0, 5002K00000yXEJkQAO, 5002K00000yWzkbQAC, 5002K00000xjFqyQAE, 5002K00000vfcGxQAI, 5002K000011f3JcQAI, 5002K000011g8DcQAI, 5006R00001lB98BQAS]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 11 Aug 2021 14:48:36 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 14 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 14 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>schwerin@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>lingzhi.deng@mongodb.com</customfieldvalue>
            <customfieldvalue>vishnu.kaushik@mongodb.com</customfieldvalue>
            <customfieldvalue>wenbin.zhu@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hzx4af:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hyu0jb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="5141">Repl 2021-08-23</customfieldvalue>
    <customfieldvalue id="5142">Repl 2021-09-06</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hzwqjb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>