<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:56:38 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-41033] set ignore_prepare=true throughout any part of index building that happens in runWithoutInterruption</title>
                <link>https://jira.mongodb.org/browse/SERVER-41033</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Currently &#160;there can be &lt;b&gt;3 way deadlock&lt;/b&gt; if step up doesn&#8217;t wait for index build to complete for the below scenario:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Node is in secondary and it starts the index build in back background process in &lt;a href=&quot;https://github.com/mongodb/mongo/blob/cf322abb3069b05e6ea32afe53a31381f0c0c6df/src/mongo/db/index_builds_coordinator.cpp#L770-L778&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;runWithoutInterruptionExceptAtGlobalShutdown&lt;/a&gt; . Let&apos;s say the index build&#8217;s plan executor currently yielded the lock.&lt;/li&gt;
	&lt;li&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/cf322abb3069b05e6ea32afe53a31381f0c0c6df/src/mongo/db/repl/replication_coordinator_impl.cpp#L1026&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Step up&lt;/a&gt;&#160; will be able to acquire the RSTL lock in mode X and finish completing its step up process.&lt;/li&gt;
	&lt;li&gt;Now, the node is in primary and index build would be able to be blocked by prepared txn due to prepare conflict. (Note: On secondaries, we have the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/cf322abb3069b05e6ea32afe53a31381f0c0c6df/src/mongo/db/repl/apply_ops.cpp#L300-L305&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;prevention mechanism&lt;/a&gt; to block transactions from being prepared if the index build is in progress. So, step up/ rollback which takes X lock and doesn&#8217;t kill operations wouldn&#8217;t get into 3 way deadlock issues).&lt;/li&gt;
	&lt;li&gt;Node tries to step down which will be blocked behind index. As the step down couldn&#8217;t kill index build for 2 reasons 1) connection is internal 2) index build is running w/ interrupt guard (runWithoutInterruptionExceptAtGlobalShutdown).&lt;/li&gt;
	&lt;li&gt;CommitTransaction cmd is waiting for RSTL lock to acquire in IX mode but blocked behind the step down thread.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;This ticket has to implement such that step up will wait for background process (index build) to complete after it has acquired the RSTL lock in X mode &lt;a href=&quot;https://github.com/mongodb/mongo/blob/f7a4c4a9632f75996ed607ffc77e2a3cab15ea88/src/mongo/db/repl/replication_coordinator_impl.cpp#L1041-L1042&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;but released the repl mutex lock&lt;/a&gt;.( like we do it in &lt;a href=&quot;https://github.com/mongodb/mongo/blob/f7a4c4a9632f75996ed607ffc77e2a3cab15ea88/src/mongo/db/repl/rollback_impl.cpp#L196-L201&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;rollback&lt;/a&gt;).&lt;/p&gt;</description>
                <environment></environment>
        <key id="759571">SERVER-41033</key>
            <summary>set ignore_prepare=true throughout any part of index building that happens in runWithoutInterruption</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="louis.williams@mongodb.com">Louis Williams</assignee>
                                    <reporter username="suganthi.mani@mongodb.com">Suganthi Mani</reporter>
                        <labels>
                    </labels>
                <created>Tue, 7 May 2019 13:46:22 +0000</created>
                <updated>Sun, 29 Oct 2023 22:21:16 +0000</updated>
                            <resolved>Wed, 29 May 2019 16:10:31 +0000</resolved>
                                                    <fixVersion>4.1.13</fixVersion>
                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>9</watches>
                                                                                                                <comments>
                            <comment id="2263897" author="xgen-internal-githook" created="Wed, 29 May 2019 16:19:09 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Louis Williams&apos;, &apos;email&apos;: &apos;louis.williams@mongodb.com&apos;, &apos;username&apos;: &apos;louiswilliams&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-41033&quot; title=&quot;set ignore_prepare=true throughout any part of index building that happens in runWithoutInterruption&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-41033&quot;&gt;&lt;del&gt;SERVER-41033&lt;/del&gt;&lt;/a&gt; Ignore prepared transactions for index builds&lt;/p&gt;

&lt;p&gt;On primaries, index builds can ignore prepare conflicts because index builds take an exclusive&lt;br/&gt;
collection lock before completing, requiring prepared transactions to commit or abort.&lt;br/&gt;
On secondaries, prepared transactions already block replication until background index builds&lt;br/&gt;
complete.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/cc1a0e5ef56838820888373c4d01e301a28f066e&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/cc1a0e5ef56838820888373c4d01e301a28f066e&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2246811" author="michael.cahill" created="Tue, 14 May 2019 23:27:45 +0000"  >&lt;p&gt;Index builds can set &lt;tt&gt;ignore_prepare=force&lt;/tt&gt; after &lt;a href=&quot;https://jira.mongodb.org/browse/WT-4580&quot; title=&quot;Abort transactions that perform updates with ignore_prepare=true &quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-4580&quot;&gt;&lt;del&gt;WT-4580&lt;/del&gt;&lt;/a&gt; is merged.&lt;/p&gt;</comment>
                            <comment id="2246625" author="louis.williams" created="Tue, 14 May 2019 20:59:33 +0000"  >&lt;p&gt;We should still be able to complete &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40177&quot; title=&quot;Enforce prepare conflicts on secondaries&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40177&quot;&gt;&lt;del&gt;SERVER-40177&lt;/del&gt;&lt;/a&gt; to enforce prepare conflicts on secondaries, and just have special behavior for index builds.&lt;/p&gt;

&lt;p&gt;This problem really only blocks &lt;a href=&quot;https://jira.mongodb.org/browse/WT-4580&quot; title=&quot;Abort transactions that perform updates with ignore_prepare=true &quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-4580&quot;&gt;&lt;del&gt;WT-4580&lt;/del&gt;&lt;/a&gt;. I think we should go ahead and complete these two SERVER tickets and reprioritize &lt;a href=&quot;https://jira.mongodb.org/browse/WT-4580&quot; title=&quot;Abort transactions that perform updates with ignore_prepare=true &quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-4580&quot;&gt;&lt;del&gt;WT-4580&lt;/del&gt;&lt;/a&gt; in the meantime.&lt;/p&gt;</comment>
                            <comment id="2246566" author="judah.schvimer" created="Tue, 14 May 2019 20:28:04 +0000"  >&lt;blockquote&gt;
&lt;p&gt;It would still be possible to block on prepare conflicts and be uninterruptible in certain phases of an index build so this may be out of the question.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;I agree since this would lead to a deadlock.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Postpone &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40177&quot; title=&quot;Enforce prepare conflicts on secondaries&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40177&quot;&gt;&lt;del&gt;SERVER-40177&lt;/del&gt;&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/browse/WT-4580&quot; title=&quot;Abort transactions that perform updates with ignore_prepare=true &quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-4580&quot;&gt;&lt;del&gt;WT-4580&lt;/del&gt;&lt;/a&gt; indefinitely.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Is there any part of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40177&quot; title=&quot;Enforce prepare conflicts on secondaries&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40177&quot;&gt;&lt;del&gt;SERVER-40177&lt;/del&gt;&lt;/a&gt; that we could do? Like could we enforce them for all operations but index builds? Could we do anything similar in &lt;a href=&quot;https://jira.mongodb.org/browse/WT-4580&quot; title=&quot;Abort transactions that perform updates with ignore_prepare=true &quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-4580&quot;&gt;&lt;del&gt;WT-4580&lt;/del&gt;&lt;/a&gt; (this seems harder)?&lt;/p&gt;</comment>
                            <comment id="2246408" author="louis.williams" created="Tue, 14 May 2019 18:53:10 +0000"  >&lt;p&gt;If we implement this fully, then this ticket will, unfortunately, directly conflict with the work for &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40177&quot; title=&quot;Enforce prepare conflicts on secondaries&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40177&quot;&gt;&lt;del&gt;SERVER-40177&lt;/del&gt;&lt;/a&gt; which would move us closer to completing &lt;a href=&quot;https://jira.mongodb.org/browse/WT-4580&quot; title=&quot;Abort transactions that perform updates with ignore_prepare=true &quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-4580&quot;&gt;&lt;del&gt;WT-4580&lt;/del&gt;&lt;/a&gt;, to return errors when performing writes with ignore_prepare=true.&lt;/p&gt;

&lt;p&gt;I see a few options:&lt;br/&gt;
1. Only ignore prepare conflicts during the collection scan phase. Re-enforce prepare conflicts during the index build drain phase to satisfy &lt;a href=&quot;https://jira.mongodb.org/browse/WT-4580&quot; title=&quot;Abort transactions that perform updates with ignore_prepare=true &quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-4580&quot;&gt;&lt;del&gt;WT-4580&lt;/del&gt;&lt;/a&gt;. It would still be possible to block on prepare conflicts and be uninterruptible in certain phases of an index build so this may be out of the question.&lt;br/&gt;
2. Perform the index build drain in separate transactions, suggested by &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.gottlieb&quot; class=&quot;user-hover&quot; rel=&quot;daniel.gottlieb&quot;&gt;daniel.gottlieb&lt;/a&gt;. This would require scanning the side table in one transaction and writing to the index with ignore_prepare=false in another. We have already dealt with numerous bugs in this specific area of code, so this is extremely risky and may be hard to get correct without serious data corruption consequences.&lt;br/&gt;
3. Postpone &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40177&quot; title=&quot;Enforce prepare conflicts on secondaries&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40177&quot;&gt;&lt;del&gt;SERVER-40177&lt;/del&gt;&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/browse/WT-4580&quot; title=&quot;Abort transactions that perform updates with ignore_prepare=true &quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-4580&quot;&gt;&lt;del&gt;WT-4580&lt;/del&gt;&lt;/a&gt; indefinitely.&lt;/p&gt;</comment>
                            <comment id="2238898" author="judah.schvimer" created="Wed, 8 May 2019 18:35:16 +0000"  >&lt;blockquote&gt;
&lt;p&gt;But, the proposal over here is to abort the transaction on primary. Am I correct?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Yes, if we blocked it, it would almost certainly get aborted at the 1 minute timeout, and not release resources in the meantime, so aborting just seems more straightforward and simpler.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Currently, rollback waits for the index build to finish. If the index build runs for day, aren&apos;t we blocking rollback too? &lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;We are blocking rollback. That&apos;s fine. While rollback is happening there is generally still a primary able to accept writes. Blocking rollback also seems more unavoidable, at least in certain cases. Rollback is relatively uncommon and a disruptive operation as well.&lt;/p&gt;</comment>
                            <comment id="2238678" author="suganthi.mani" created="Wed, 8 May 2019 16:00:55 +0000"  >&lt;blockquote&gt;&lt;p&gt;Instead of blocking stepup on background index builds completing, abort transactions at prepare time if there is a background index build built on a collection included in the transaction. This would be comparable to what we do for secondaries at prepare time now in&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-38588&quot; title=&quot;Hybrid index builds do not work when applied concurrently with prepared transactions on secondaries&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-38588&quot;&gt;&lt;del&gt;SERVER-38588&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;In &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-38588&quot; title=&quot;Hybrid index builds do not work when applied concurrently with prepared transactions on secondaries&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-38588&quot;&gt;&lt;del&gt;SERVER-38588&lt;/del&gt;&lt;/a&gt;, we block prepare transaction on secondaries if background index build is running on a collection included in the transaction. But, the proposal over here is to abort the transaction on primary. Am I correct?&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;Set ignore_prepare=true throughout all of index building.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;This means, we won&apos;t hit prepare conflict for index build that was started during secondary state and got bled into primary. So, when the primary steps down, we still won&apos;t be able kill the index build (due to internal operation and runWithoutInterruption). But, we will wait for query executor to yield the IX locks (similar to read operation) for step down to continue. So, this solution fixes the dead lock and sounds reasonable.&lt;/p&gt;

&lt;p&gt;Currently, rollback waits for the index build to finish. If the index build runs for day, aren&apos;t we blocking rollback too? &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=judah.schvimer&quot; class=&quot;user-hover&quot; rel=&quot;judah.schvimer&quot;&gt;judah.schvimer&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2238638" author="judah.schvimer" created="Wed, 8 May 2019 15:29:59 +0000"  >&lt;p&gt;After discussing with &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=milkie&quot; class=&quot;user-hover&quot; rel=&quot;milkie&quot;&gt;milkie&lt;/a&gt;, there are two ways we could go about this.&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Instead of blocking stepup on background index builds completing, abort transactions at prepare time if there is a background index build built on a collection included in the transaction. This would be comparable to what we do for secondaries at prepare time now in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-38588&quot; title=&quot;Hybrid index builds do not work when applied concurrently with prepared transactions on secondaries&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-38588&quot;&gt;&lt;del&gt;SERVER-38588&lt;/del&gt;&lt;/a&gt;.&lt;/li&gt;
	&lt;li&gt;Set &lt;tt&gt;ignore_prepare=true&lt;/tt&gt; throughout all of index building. &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.gottlieb&quot; class=&quot;user-hover&quot; rel=&quot;daniel.gottlieb&quot;&gt;daniel.gottlieb&lt;/a&gt; thinks this is essentially required and &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=milkie&quot; class=&quot;user-hover&quot; rel=&quot;milkie&quot;&gt;milkie&lt;/a&gt; thinks it is safe. &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=suganthi.mani&quot; class=&quot;user-hover&quot; rel=&quot;suganthi.mani&quot;&gt;suganthi.mani&lt;/a&gt;, do you agree this would fix the deadlock in this ticket?&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;I&apos;m repurposing this ticket to be &quot;set ignore_prepare=true throughout any part of index building that happens in runWithoutInterruption&quot; and sending to the storage team.&lt;/p&gt;</comment>
                            <comment id="2238535" author="judah.schvimer" created="Wed, 8 May 2019 14:37:02 +0000"  >&lt;p&gt;Looking at this now, this behavior would be undesirable since it could prevent a primary from getting elected for over a day. I&apos;m exploring other solutions.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                        <issuelink>
            <issuekey id="759582">SERVER-41034</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="740577">SERVER-40700</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="999011">SERVER-44577</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="787591">SERVER-41462</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="971168">SERVER-44045</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 8 May 2019 14:37:02 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 37 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-1032</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 37 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>judah.schvimer@mongodb.com</customfieldvalue>
            <customfieldvalue>louis.williams@mongodb.com</customfieldvalue>
            <customfieldvalue>michael.cahill@mongodb.com</customfieldvalue>
            <customfieldvalue>suganthi.mani@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|huy3nr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr78mf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2909">Storage NYC 2019-05-20</customfieldvalue>
    <customfieldvalue id="2983">Execution Team 2019-06-03</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|huxpx3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>