<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:47:38 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-37988] recover locks on step up at the beginning of the state transition rather than at the end</title>
                <link>https://jira.mongodb.org/browse/SERVER-37988</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Prepared transactions hold collection and database IX locks across state transitions, and can only release those locks by processing a &apos;commitTransaction&apos; or &apos;abortTransaction&apos; command. It is very possible that in order to receive a &apos;commitTransaction&apos; or &apos;abortTransaction&apos; command, a node must go through a state transition to step up or down. Thus we cannot just wait for the prepared transaction to release its locks to acquire locks for the state transition.&lt;/p&gt;

&lt;p&gt;This means that state transitions cannot acquire locks that would conflict with locks held by a prepared transaction.&lt;/p&gt;

&lt;p&gt;The only state transition that I know of that acquires locks is step-up.&lt;/p&gt;

&lt;p&gt;Shutdown does not count as a state transition. It aborts all prepared storage transactions without making that abort durable in any way before acquiring any locks. This occurs after shutting down the replication system (though before &lt;a href=&quot;https://github.com/mongodb/mongo/blob/6e8f2a7109505385d902b017112433a133eff9d5/src/mongo/db/db.cpp#L908&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;killing all operations&lt;/a&gt;), so I think it should be safe.&lt;/p&gt;</description>
                <environment></environment>
        <key id="630322">SERVER-37988</key>
            <summary>recover locks on step up at the beginning of the state transition rather than at the end</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="4" iconUrl="https://jira.mongodb.org/images/icons/priorities/minor.svg">Minor - P4</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="gregory.wlodarek@mongodb.com">Gregory Wlodarek</assignee>
                                    <reporter username="judah.schvimer@mongodb.com">Judah Schvimer</reporter>
                        <labels>
                            <label>txn_storage</label>
                    </labels>
                <created>Wed, 7 Nov 2018 19:12:01 +0000</created>
                <updated>Sun, 29 Oct 2023 22:26:47 +0000</updated>
                            <resolved>Wed, 15 May 2019 19:38:03 +0000</resolved>
                                                    <fixVersion>4.1.12</fixVersion>
                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>8</watches>
                                                                                                                <comments>
                            <comment id="2247914" author="xgen-internal-githook" created="Wed, 15 May 2019 19:37:53 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Gregory Wlodarek&apos;, &apos;username&apos;: &apos;GWlodarek&apos;, &apos;email&apos;: &apos;gregory.wlodarek@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-37988&quot; title=&quot;recover locks on step up at the beginning of the state transition rather than at the end&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-37988&quot;&gt;&lt;del&gt;SERVER-37988&lt;/del&gt;&lt;/a&gt; Recover locks on step up at the beginning of the state transition rather than at the end&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/bae6f255cb2c99d557f02d3861c4abb654fe3071&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/bae6f255cb2c99d557f02d3861c4abb654fe3071&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2247912" author="xgen-internal-githook" created="Wed, 15 May 2019 19:36:42 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;email&apos;: &apos;gregory.wlodarek@mongodb.com&apos;, &apos;name&apos;: &apos;Gregory Wlodarek&apos;, &apos;username&apos;: &apos;GWlodarek&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-37988&quot; title=&quot;recover locks on step up at the beginning of the state transition rather than at the end&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-37988&quot;&gt;&lt;del&gt;SERVER-37988&lt;/del&gt;&lt;/a&gt; Add an optional predicate argument in forEachCollectionFromDb() to allow skipping unsatisfied collections without grabbing them in the desired lock mode&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/9af1defd2b1b831322e716baa8a4e54b27bd534a&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/9af1defd2b1b831322e716baa8a4e54b27bd534a&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2247895" author="xgen-internal-githook" created="Wed, 15 May 2019 19:27:44 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;email&apos;: &apos;gregory.wlodarek@mongodb.com&apos;, &apos;name&apos;: &apos;Gregory Wlodarek&apos;, &apos;username&apos;: &apos;GWlodarek&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-37988&quot; title=&quot;recover locks on step up at the beginning of the state transition rather than at the end&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-37988&quot;&gt;&lt;del&gt;SERVER-37988&lt;/del&gt;&lt;/a&gt; Change Collection::getIndexSize() to be const&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/9054590882e5fb7b205593e100e83b17f85b9108&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/9054590882e5fb7b205593e100e83b17f85b9108&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2231092" author="judah.schvimer" created="Wed, 1 May 2019 20:36:44 +0000"  >&lt;p&gt;We will continue yielding locks for the time being, so this ticket is still required.&lt;/p&gt;</comment>
                            <comment id="2222570" author="judah.schvimer" created="Tue, 23 Apr 2019 21:26:06 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40723&quot; title=&quot;Deadlock between S lock acquisition on secondary and prepare conflict&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40723&quot;&gt;&lt;del&gt;SERVER-40723&lt;/del&gt;&lt;/a&gt; may stop yielding locks for prepared transactions which will make this ticket go away.&lt;/p&gt;</comment>
                            <comment id="2217611" author="judah.schvimer" created="Thu, 18 Apr 2019 12:16:55 +0000"  >&lt;p&gt;Sorry for the confusing question. I was unclear on why, today, we kill global S locks on step down but not step up. I was asking if this ticket was related to why we didn&apos;t have to kill global S locks on step up today. I&apos;ve asked this on &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40700&quot; title=&quot;Deadlock between read prepare conflicts and state transitions&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40700&quot;&gt;&lt;del&gt;SERVER-40700&lt;/del&gt;&lt;/a&gt; in response to Suganthi&apos;s comment and will move the discussion there.&lt;/p&gt;</comment>
                            <comment id="2217268" author="tess.avitabile" created="Wed, 17 Apr 2019 22:57:19 +0000"  >&lt;p&gt;I&apos;m not sure I understand the question. I think the deadlock described in&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40700&quot; title=&quot;Deadlock between read prepare conflicts and state transitions&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40700&quot;&gt;&lt;del&gt;SERVER-40700&lt;/del&gt;&lt;/a&gt; can also happen on step-up (prepared transaction waits for state transition; afterClusterTime read waits for prepared transaction; stepup waits for afterClusterTime read), but it will be fixed in the same way as stepdown, by making sure that reads don&apos;t conflict with stepup/stepdown. However, the original problem of dropping temp collections preventing us from recovering locks at the beginning of state transition still exists.&lt;/p&gt;</comment>
                            <comment id="2217152" author="judah.schvimer" created="Wed, 17 Apr 2019 21:12:34 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tess.avitabile&quot; class=&quot;user-hover&quot; rel=&quot;tess.avitabile&quot;&gt;tess.avitabile&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=suganthi.mani&quot; class=&quot;user-hover&quot; rel=&quot;suganthi.mani&quot;&gt;suganthi.mani&lt;/a&gt;, will this cause us to have to kill operations that acquire global S locks on step-up, similar to what we do on step-down? Or does this go away with locking changes in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40700&quot; title=&quot;Deadlock between read prepare conflicts and state transitions&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40700&quot;&gt;&lt;del&gt;SERVER-40700&lt;/del&gt;&lt;/a&gt;?&lt;/p&gt;</comment>
                            <comment id="2200818" author="milkie" created="Wed, 3 Apr 2019 17:20:08 +0000"  >&lt;p&gt;By moving the lock recovery from the end of the stepup to the beginning, we hope to discover any further lock violations; these will manifest as deadlocks.&lt;/p&gt;</comment>
                            <comment id="2157606" author="judah.schvimer" created="Wed, 20 Feb 2019 20:52:00 +0000"  >&lt;p&gt;I wanted to signal (mostly to myself) that this was lower priority in the prepare epic, but still required. This seemed like the best way. I can change it back if you&apos;d prefer not to (somewhat) overload the meaning of the field.&lt;/p&gt;</comment>
                            <comment id="2157572" author="tess.avitabile" created="Wed, 20 Feb 2019 20:31:59 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=judah.schvimer&quot; class=&quot;user-hover&quot; rel=&quot;judah.schvimer&quot;&gt;judah.schvimer&lt;/a&gt;, why was the priority reduced to minor?&lt;/p&gt;</comment>
                            <comment id="2062461" author="judah.schvimer" created="Wed, 14 Nov 2018 22:22:25 +0000"  >&lt;p&gt;Based on this discussion and talking to &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tess.avitabile&quot; class=&quot;user-hover&quot; rel=&quot;tess.avitabile&quot;&gt;tess.avitabile&lt;/a&gt;, we will fix this in  &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-37199&quot; title=&quot;Yield locks of transactions in secondary application&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-37199&quot;&gt;&lt;del&gt;SERVER-37199&lt;/del&gt;&lt;/a&gt; (option 3) for now. This ticket will then become &quot;recover locks on step up at the beginning of the state transition rather than at the end&quot; when the Replication or Storage teams have time to fix collection drops to only take collection X locks.&lt;/p&gt;</comment>
                            <comment id="2062325" author="geert.bosch" created="Wed, 14 Nov 2018 20:58:47 +0000"  >&lt;p&gt;While we&apos;re very close to being able to do option 2), we are not completely there yet. On the positive side, all (temporary) collections that need dropping on step up will be 2-phase dropped, so in essence they only get a name change. After the patch to no longer destroy Collection objects on rename, we don&apos;t need the &lt;tt&gt;MODE_X&lt;/tt&gt;&#160;database lock to prevent Collection pointers from becoming invalidated. However, the one remaining thing is that we need to update the name in the Database object, and that is protected by the database lock.&lt;/p&gt;

&lt;p&gt;So maybe we should go with option 3 for now.&lt;/p&gt;</comment>
                            <comment id="2062230" author="judah.schvimer" created="Wed, 14 Nov 2018 19:58:53 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=geert.bosch&quot; class=&quot;user-hover&quot; rel=&quot;geert.bosch&quot;&gt;geert.bosch&lt;/a&gt;, do you know how big of a task 2) is?&lt;/p&gt;

&lt;p&gt;After talking with &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=milkie&quot; class=&quot;user-hover&quot; rel=&quot;milkie&quot;&gt;milkie&lt;/a&gt;, there appears to be an option 3.&lt;br/&gt;
3) Release prepared transaction locks on secondaries as soon as they successfully prepare (described &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-37199?focusedCommentId=2053781&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-2053781&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;). Only restore the prepared locks &lt;b&gt;after&lt;/b&gt; we do all of the transition work we must do. This means that if we do any writes during the transition that should have conflicted with the prepared transaction, even taking as fine-grained locks as possible, then instead of deadlocking we will potentially corrupt data. &lt;/p&gt;</comment>
                            <comment id="2062221" author="geert.bosch" created="Wed, 14 Nov 2018 19:51:36 +0000"  >&lt;p&gt;I much prefer 2). AFAIK we already ban temp collections in transactions. We want to do the second part regardless for many reasons. Given that wouldn&apos;t 2) be both easier and safer?&lt;/p&gt;</comment>
                            <comment id="2062166" author="judah.schvimer" created="Wed, 14 Nov 2018 19:29:20 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=geert.bosch&quot; class=&quot;user-hover&quot; rel=&quot;geert.bosch&quot;&gt;geert.bosch&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=milkie&quot; class=&quot;user-hover&quot; rel=&quot;milkie&quot;&gt;milkie&lt;/a&gt;, I currently have two plans for fixing the temp collection locking. Please let me know your thoughts or ideas.&lt;br/&gt;
1) Asynchronously drop temp collections after step up. Change the temp collection names to use UUIDs so they are not used and do not cause collisions after step up and simply schedule a task to drop all of the temp collections that will succeed as soon as it can acquire the required locks. &lt;br/&gt;
2) Ban temp collections in transactions and make collection drops (or temp collection drops specifically) only acquire a collection lock in mode X and a database lock in mode IX. These drops are two phase drops, so they&apos;re really renames within a database. I don&apos;t know how we&apos;d change the locking requirements here.&lt;/p&gt;

&lt;p&gt;1) seems easier but 2) feels safer and easier to know it&apos;s correct.&lt;/p&gt;</comment>
                            <comment id="2061961" author="judah.schvimer" created="Wed, 14 Nov 2018 17:20:05 +0000"  >&lt;p&gt;After discussing with &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=charlie.swanson&quot; class=&quot;user-hover&quot; rel=&quot;charlie.swanson&quot;&gt;charlie.swanson&lt;/a&gt;, agg $out and map-reduce both use temporary collections on arbitrary databases. Those collections do not need to be dropped synchronously on step up. If they&apos;re done asynchronously we must make sure that we only drop the temp collections from before the step up and not any new ones, and that no reads and writes go to the temp collections after step up. Changing temporary collection names (which now use an atomic counter but could use a UUID for uniqueness) to not collide would solve this problem. This ticket will need to make dropping those temp collections not conflict with locks held by prepared transactions.&lt;/p&gt;</comment>
                            <comment id="2060921" author="judah.schvimer" created="Tue, 13 Nov 2018 23:17:42 +0000"  >&lt;p&gt;Here is a list of the S and X locks taken on step up:&lt;br/&gt;
1) Sharding initializes collections in the &apos;config&apos; database, which is not allowed in transactions currently.&lt;br/&gt;
2) ShardingStateRecovery acquires a collection X lock on &apos;admin.system.version&apos; which cannot be used in a transaction.&lt;br/&gt;
3) Sharding updates the shard identity config string and acquires a collection X lock on &apos;admin.system.version&apos; which cannot be used in a transaction.&lt;br/&gt;
4) We initialize the session catalog, by creating the &apos;config.transactions&apos; collection. This is in the &apos;config&apos; database, which is not allowed in transactions currently.&lt;br/&gt;
5) We drop all temp collections. This acquires a database X lock for all databases that have a temp collection. &lt;/p&gt;

&lt;p&gt;The only work for this ticket is to fix temp collections or decide they are fine as is. I do not think we need to support temporary collections in transactions, though work in this ticket would need to ban them from transactions if we needed to take a collection lock in mode X on them.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="630321">SERVER-37987</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="634060">SERVER-38139</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="634068">SERVER-38140</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="687942">SERVER-39520</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="740577">SERVER-40700</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="741394">SERVER-40723</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="611497">SERVER-37381</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="553260">SERVER-35365</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="606934">SERVER-37199</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="759654">SERVER-41037</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>18.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 14 Nov 2018 19:51:36 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 39 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/SERVER-37987'>SERVER-37987</a></s>, <s><a href='https://jira.mongodb.org/browse/SERVER-38139'>SERVER-38139</a></s>, <s><a href='https://jira.mongodb.org/browse/SERVER-38140'>SERVER-38140</a></s>, <s><a href='https://jira.mongodb.org/browse/SERVER-39520'>SERVER-39520</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-1032</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 39 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>milkie@mongodb.com</customfieldvalue>
            <customfieldvalue>geert.bosch@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>gregory.wlodarek@mongodb.com</customfieldvalue>
            <customfieldvalue>judah.schvimer@mongodb.com</customfieldvalue>
            <customfieldvalue>tess.avitabile@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|huca4v:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr78sv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2908">Storage NYC 2019-05-06</customfieldvalue>
    <customfieldvalue id="2909">Storage NYC 2019-05-20</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hubwe7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>