<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:17:37 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-48617] Concurrent &apos;createIndexes&apos; can block all write operations on a primary and standalone by exhausting write tickets.</title>
                <link>https://jira.mongodb.org/browse/SERVER-48617</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We can get into below 2 way deadlock which will block all write operations on a primary and standalone.&lt;br/&gt;
 1)&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/4cf4600c81684ceb7f07facf64a1ba01519d3dae/src/mongo/db/commands/create_indexes.cpp#L462&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&apos;createIndexes&apos; cmd&lt;/a&gt;&#160;user threads&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/4cf4600c81684ceb7f07facf64a1ba01519d3dae/src/mongo/db/index_builds_coordinator_mongod.cpp#L144&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;waits on a conditional variable&lt;/a&gt;&#160;for indexBuildCoordinator worker threads to get freed up, by holding&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/4cf4600c81684ceb7f07facf64a1ba01519d3dae/src/mongo/db/index_builds_coordinator_mongod.cpp#L136&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;global lock&lt;/a&gt;&#160;in IX mode.&#160;When global lock is in IX mode is acquired, we&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/4cf4600c81684ceb7f07facf64a1ba01519d3dae/src/mongo/db/concurrency/lock_state.cpp#L376-L391&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;implicitly&lt;/a&gt;&#160;acquire a write ticket. So, those &apos;createIndexes&apos; threads&#160;are waiting for indexBuildsCoordinator worker thread to get freed up by holding write tickets and eventually exhausting those write tickets.&lt;/p&gt;

&lt;p&gt;2)&#160;IndexBuildsCoordinator worker threads are waiting for write tickets (For e.g., when restoring yielded locks during &lt;a href=&quot;https://github.com/mongodb/mongo/blob/4cf4600c81684ceb7f07facf64a1ba01519d3dae/src/mongo/db/index_builds_coordinator.cpp#L2102&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;collection scan phase&lt;/a&gt;) for index completion.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1373576">SERVER-48617</key>
            <summary>Concurrent &apos;createIndexes&apos; can block all write operations on a primary and standalone by exhausting write tickets.</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="benety.goh@mongodb.com">Benety Goh</assignee>
                                    <reporter username="suganthi.mani@mongodb.com">Suganthi Mani</reporter>
                        <labels>
                    </labels>
                <created>Fri, 5 Jun 2020 15:15:25 +0000</created>
                <updated>Sun, 29 Oct 2023 22:07:22 +0000</updated>
                            <resolved>Fri, 10 Jul 2020 01:22:16 +0000</resolved>
                                                    <fixVersion>4.7.0</fixVersion>
                    <fixVersion>4.4.2</fixVersion>
                                    <component>Index Maintenance</component>
                                        <votes>0</votes>
                                    <watches>5</watches>
                                                                                                                <comments>
                            <comment id="3380722" author="xgen-internal-githook" created="Tue, 8 Sep 2020 19:50:01 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Benety Goh&apos;, &apos;email&apos;: &apos;benety@mongodb.com&apos;, &apos;username&apos;: &apos;benety&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-48617&quot; title=&quot;Concurrent &amp;#39;createIndexes&amp;#39; can block all write operations on a primary and standalone by exhausting write tickets.&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-48617&quot;&gt;&lt;del&gt;SERVER-48617&lt;/del&gt;&lt;/a&gt; do not hold global lock while waiting for index build thread resource&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 7afcf9895eb7da296f602604e2973cdb6fa0c67f)&lt;br/&gt;
Branch: v4.4&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/d9227bf6bbcc1918c954ef2485e12157e2349fbf&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/d9227bf6bbcc1918c954ef2485e12157e2349fbf&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3278244" author="xgen-internal-githook" created="Thu, 9 Jul 2020 19:08:22 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Benety Goh&apos;, &apos;email&apos;: &apos;benety@mongodb.com&apos;, &apos;username&apos;: &apos;benety&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-48617&quot; title=&quot;Concurrent &amp;#39;createIndexes&amp;#39; can block all write operations on a primary and standalone by exhausting write tickets.&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-48617&quot;&gt;&lt;del&gt;SERVER-48617&lt;/del&gt;&lt;/a&gt; do not hold global lock while waiting for index build thread resource&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/7afcf9895eb7da296f602604e2973cdb6fa0c67f&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/7afcf9895eb7da296f602604e2973cdb6fa0c67f&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3275473" author="suganthi.mani" created="Wed, 8 Jul 2020 18:16:12 +0000"  >&lt;p&gt;Just want to record some offline conversation we had it in the slack.&lt;br/&gt;
Skipping ticket acquisition won&apos;t be an ideal solution for this ticket. Because we can still get into deadlock scenario if we hit this sequence. &lt;br/&gt;
1) create index thread waits on CV holding global lock in X mode for index builder thread to freed up.&lt;br/&gt;
2) Some&#160; operation (XXXX???) tries to acquire global lock in X mode and so enqueue the&#160; global lock and stuck behind the createIndexes thread&lt;br/&gt;
3) Index build thread tries to acquire global lock in doing index build work (collection scan,&#160; draining &amp;amp; commit phases)&#160; and stuck behind the some operation (XXXX???)&lt;br/&gt;
So, it&apos;s always not safe to hold any locks while holding the mutex lock.&lt;/p&gt;</comment>
                            <comment id="3275452" author="benety.goh" created="Wed, 8 Jul 2020 18:06:09 +0000"  >&lt;p&gt;As follow-up work, we would like to look into detecting and preventing other instances of the resource locking/condition variable anti-pattern in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-49369&quot; title=&quot;add check for Global resource lock state to OperationContext::waitForConditionOrInterrupt()&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-49369&quot;&gt;&lt;del&gt;SERVER-49369&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="3275062" author="benety.goh" created="Wed, 8 Jul 2020 15:39:51 +0000"  >&lt;p&gt;The code that waits on the conditional variable while holding the global lock was introduced in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-47155&quot; title=&quot;Limit the number of simultaneous index builds running from user connections to 3&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-47155&quot;&gt;&lt;del&gt;SERVER-47155&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="3205220" author="suganthi.mani" created="Thu, 11 Jun 2020 15:08:06 +0000"  >&lt;blockquote&gt;
&lt;p&gt;I don&apos;t think the extra waiting is very important because it won&apos;t block while holding onto locks or storage resources. I think this is a rare enough case that the extra waiting is not critical.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;You are right, extra waiting is not critical unless if we don&apos;t hold any mongoDB locks /or/ storage resources . If we go by this approach, I would expect to have some invariant check to verify opCtx is not holding &lt;a href=&quot;https://github.com/mongodb/mongo/blob/19656470dc5e1ce2229e89ba701f0ec708e69072/src/mongo/db/concurrency/lock_state.cpp#L175&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;any locks&lt;/a&gt; (or just more specific &lt;a href=&quot;https://github.com/mongodb/mongo/blob/19656470dc5e1ce2229e89ba701f0ec708e69072/src/mongo/db/concurrency/lock_state.cpp#L191&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;RSTL lock&lt;/a&gt;) before CV wait. This is to make sure we don&apos;t break the contract in future and get into deadlock scenarios.&lt;/p&gt;

&lt;p&gt;But, my design approach would be stepdown should interrupt all write operations &lt;b&gt;and make the cmd response available as quick as possible to the user.&lt;/b&gt; And, I consider createIndexes as a write operation thread. Though I agree many concurrent index creation in real world is rare. Anyways, I am ok with with either approach.&lt;/p&gt;

</comment>
                            <comment id="3205079" author="louis.williams" created="Thu, 11 Jun 2020 14:20:50 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=suganthi.mani&quot; class=&quot;user-hover&quot; rel=&quot;suganthi.mani&quot;&gt;suganthi.mani&lt;/a&gt;, my question was not directed at just the Global lock. Why do we need to be holding the RSTL, or any lock, while waiting on a condition variable?&lt;/p&gt;

&lt;p&gt;edited based on your update: Yes, we should not be holding locks while waiting on the condition. I don&apos;t think the extra waiting is very important because it won&apos;t block while holding onto locks or storage resources. I think this is a rare enough case that the extra waiting is not critical.&lt;/p&gt;</comment>
                            <comment id="3205040" author="suganthi.mani" created="Thu, 11 Jun 2020 14:09:53 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=louis.williams&quot; class=&quot;user-hover&quot; rel=&quot;louis.williams&quot;&gt;louis.williams&lt;/a&gt;  If the thread is waiting on CV just by holding RSTL lock, then it would block step down which requires RSTL in X mode. Stepdown only interrupts/kills thread holding global lock in IX, X or S mode. Currently we support only 2 modes for RSTL.  All read/write operations acquire RSTL in IX mode and repl state transition  threads acquire RSTL in X mode.  Stepdown needs to kill only only write operation and not read operations. And, that information can be obtained only via global locks.&lt;/p&gt;

&lt;p&gt;The other option is not to hold any locks while waiting for &lt;a href=&quot;https://github.com/mongodb/mongo/blob/4cf4600c81684ceb7f07facf64a1ba01519d3dae/src/mongo/db/index_builds_coordinator_mongod.cpp#L144&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this conditional wait&lt;/a&gt;. Only, downside is that the node might keep waiting on CV even after the node transitioned to secondary state. But, if you are going with this approach, we should be careful about the locking order when checking the repl state, it should be MongoDB locks (global or RSTL lock) followed by the indexBuildsCoordinator &lt;a href=&quot;https://github.com/mongodb/mongo/blob/4cf4600c81684ceb7f07facf64a1ba01519d3dae/src/mongo/db/index_builds_coordinator_mongod.cpp#L138&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;mutex&lt;/a&gt; lock. &lt;/p&gt;</comment>
                            <comment id="3204998" author="louis.williams" created="Thu, 11 Jun 2020 13:57:02 +0000"  >&lt;p&gt;It seems like the problem is that we are holding a Global IX lock while &lt;a href=&quot;https://github.com/mongodb/mongo/blob/4cf4600c81684ceb7f07facf64a1ba01519d3dae/src/mongo/db/index_builds_coordinator_mongod.cpp#L144&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;blocking on a condition variable&lt;/a&gt;, which is an antipattern that leads to deadlocks.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=suganthi.mani&quot; class=&quot;user-hover&quot; rel=&quot;suganthi.mani&quot;&gt;suganthi.mani&lt;/a&gt; you said:&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;To be noted, we acquire global lock in IX instead of directly acquiring RSTL lock in IX mode before waiting on this cv is because we want the conditional variable wait to be interrupted on step down. Stepdown thread will interrupt only threads holding global lock in IX, X and S mode.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;Why does the operation need to be immediately interrupted by stepdown? If it&apos;s not holding a global lock, then it&apos;s not holding up any resources and doesn&apos;t need to be interrupted immediately. When a threadpool resource is eventually freed, the operation will discover that it is no longer primary and just return an error.&lt;/p&gt;</comment>
                            <comment id="3192675" author="suganthi.mani" created="Fri, 5 Jun 2020 16:34:45 +0000"  >&lt;p&gt;Another important point is that, it seems when we set skip acquiring ticket on opCtx we can&apos;t reset the &apos;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/4cf4600c81684ceb7f07facf64a1ba01519d3dae/src/mongo/db/concurrency/locker.h#L488&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;_shouldAcquireTicket&lt;/a&gt;&apos;. CreateIndexes user thread sometimes have to abort the index build  which will need it to write the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/4cf4600c81684ceb7f07facf64a1ba01519d3dae/src/mongo/db/commands/create_indexes.cpp#L638&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;abortIndexBuild oplog entry&lt;/a&gt;. Though, writing oplog is a WT write operation, we can skip the ticketing mechanism by applying &lt;a href=&quot;https://github.com/mongodb/mongo/blob/4cf4600c81684ceb7f07facf64a1ba01519d3dae/src/mongo/db/transaction_participant.cpp#L983-L985&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this&lt;/a&gt; reasoning that we used in transaction code path.&lt;/p&gt;</comment>
                            <comment id="3192625" author="suganthi.mani" created="Fri, 5 Jun 2020 16:14:12 +0000"  >&lt;p&gt;Easiest solution is that we can make &apos;createIndexes&apos; user threads to skip acquiring write ticket when acquiring &lt;a href=&quot;https://github.com/mongodb/mongo/blob/4cf4600c81684ceb7f07facf64a1ba01519d3dae/src/mongo/db/index_builds_coordinator_mongod.cpp#L136&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;global lock&lt;/a&gt; by using &lt;a href=&quot;https://github.com/mongodb/mongo/blob/4cf4600c81684ceb7f07facf64a1ba01519d3dae/src/mongo/db/concurrency/locker.h#L485&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;skipAcquireTicket()&lt;/a&gt;. The goal of ticketing mechanism is to throttle number of concurrent WT write/read operations. &apos;CreateIndexes&apos; user (parent) threads are not going make any WT write operations, only indexBuildsCoordinator worker threads are going to build index and perform WT index table write operations. So, it&apos;s ok for &apos;createIndexes&apos; cmd user threads to skip acquiring ticket when acquiring global lock at &lt;a href=&quot;https://github.com/mongodb/mongo/blob/4cf4600c81684ceb7f07facf64a1ba01519d3dae/src/mongo/db/index_builds_coordinator_mongod.cpp#L136&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this&lt;/a&gt; point.&lt;/p&gt;

&lt;p&gt;To be noted, we acquire global lock in IX instead of directly acquiring RSTL lock in IX mode before waiting on &lt;a href=&quot;https://github.com/mongodb/mongo/blob/4cf4600c81684ceb7f07facf64a1ba01519d3dae/src/mongo/db/index_builds_coordinator_mongod.cpp#L144&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this cv&lt;/a&gt; is because we want the conditional variable wait to be interrupted on step down. Stepdown thread will interrupt only threads holding global lock in IX, X and S mode.&lt;/p&gt;

&lt;p&gt;It also seems collection scanning phase and drain phase acquire write tickets because we acquire &lt;a href=&quot;https://github.com/mongodb/mongo/blob/4cf4600c81684ceb7f07facf64a1ba01519d3dae/src/mongo/db/index_builds_coordinator.cpp#L2102&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;DB lock in IX&lt;/a&gt; mode. So, do we really need DB and collection lock in IX mode? can we make it as IS mode? To be noted, this won&apos;t fix the deadlock issue, just nit. Seems historically, collection scan phase &amp;amp; drain phase used to acquire DB lock in IX mode and &lt;a href=&quot;https://github.com/mongodb/mongo/blame/76d4548a751a56c8faf1887114685b540203a650/src/mongo/db/index_builds_coordinator.cpp#L2038&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;collection in IS or S&lt;/a&gt; mode.&lt;br/&gt;
&#160;&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="1294175">SERVER-47155</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1404219">SERVER-49369</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>11.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="18953"><![CDATA[v4.4]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 11 Jun 2020 13:57:02 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 22 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 22 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>benety.goh@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>louis.williams@mongodb.com</customfieldvalue>
            <customfieldvalue>suganthi.mani@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxoq53:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hxbv8v:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3938">Execution Team 2020-06-29</customfieldvalue>
    <customfieldvalue id="3939">Execution Team 2020-07-13</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxocef:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>