<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:16:13 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-48125] Stepdown can deadlock with storing lastVote via journal flusher</title>
                <link>https://jira.mongodb.org/browse/SERVER-48125</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;As part of storing the lastVote document, we will wait for it to be durable, and we will eventually call into &lt;a href=&quot;https://github.com/mongodb/mongo/blob/3de4762514cee3507f77efd7b46b20c4362c041a/src/mongo/db/repl/replication_consistency_markers_impl.cpp#L461&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;refreshOplogTruncateAfterPointIfPrimary&lt;/a&gt;. This needs to acquire a global IX lock as part of an &lt;a href=&quot;https://github.com/mongodb/mongo/blob/3de4762514cee3507f77efd7b46b20c4362c041a/src/mongo/db/repl/replication_consistency_markers_impl.cpp#L482&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;AutoGetCollection&lt;/a&gt;. This can deadlock with stepdown, as it tries to &lt;a href=&quot;https://github.com/mongodb/mongo/blob/3de4762514cee3507f77efd7b46b20c4362c041a/src/mongo/db/repl/replication_coordinator_external_state_impl.cpp#L781&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;clear&lt;/a&gt; the oplog truncate after point, which in turn waits on a journal flush. The journal flusher needs to be able to run wait for durability too, but it cannot get to the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/be4fd3cca7fc4ea6883ceb1e262b0ac5cc7ecf69/src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp#L315&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;critical section&lt;/a&gt; as that is protected by a mutex which is already held by the lastVote thread.&lt;/p&gt;

&lt;p&gt;We recently made storing the lastVote document fully uninterruptible in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-47612&quot; title=&quot;Elections not robust in remove_newly_added_field_after_finishing_initial_sync.js&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-47612&quot;&gt;&lt;del&gt;SERVER-47612&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1347624">SERVER-48125</key>
            <summary>Stepdown can deadlock with storing lastVote via journal flusher</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="backlog-server-repl">Backlog - Replication Team</assignee>
                                    <reporter username="vesselina.ratcheva@mongodb.com">Vesselina Ratcheva</reporter>
                        <labels>
                    </labels>
                <created>Tue, 12 May 2020 05:15:15 +0000</created>
                <updated>Tue, 6 Dec 2022 02:27:23 +0000</updated>
                            <resolved>Wed, 20 May 2020 22:35:48 +0000</resolved>
                                                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="3098502" author="dianna.hohensee" created="Wed, 20 May 2020 22:29:38 +0000"  >&lt;p&gt;Yes! This ticket should be fixed by &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-48144&quot; title=&quot;waitUntilDurable should not take a mutex before taking locks&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-48144&quot;&gt;&lt;del&gt;SERVER-48144&lt;/del&gt;&lt;/a&gt;&apos;s commit and can now be closed.&lt;/p&gt;</comment>
                            <comment id="3098449" author="judah.schvimer" created="Wed, 20 May 2020 21:54:37 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=dianna.hohensee&quot; class=&quot;user-hover&quot; rel=&quot;dianna.hohensee&quot;&gt;dianna.hohensee&lt;/a&gt;, is it safe for me to close this ticket as a duplicate of  &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-48144&quot; title=&quot;waitUntilDurable should not take a mutex before taking locks&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-48144&quot;&gt;&lt;del&gt;SERVER-48144&lt;/del&gt;&lt;/a&gt;, or is there more work to do here?&lt;/p&gt;</comment>
                            <comment id="3081904" author="dianna.hohensee" created="Tue, 12 May 2020 20:35:39 +0000"  >&lt;p&gt;We will have to add some kind of retryability on stepdown interrupt to JournalFlusher::waitForJournalFlush in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-48149&quot; title=&quot;Move callers of waitUntilDurable onto JournalFlusher::waitForJournalFlush&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-48149&quot;&gt;&lt;del&gt;SERVER-48149&lt;/del&gt;&lt;/a&gt;, so that callers can eventually succeed.&lt;/p&gt;

&lt;p&gt;Stepdown toggles off updating the oplogTruncateAfterPoint, then makes sure a round of flushing finishes in the JournalFlusher to clear the system before at last unsetting the oplogTruncateAfterPoint. So retrying immediately after that is safe: journal flush without the no longer needed oplogTruncateAfterPoint update that is causing the problems. It&apos;s just the race of waitUntilDurable callers not going through the JournalFlusher.&lt;/p&gt;</comment>
                            <comment id="3081852" author="judah.schvimer" created="Tue, 12 May 2020 20:14:45 +0000"  >&lt;blockquote&gt;
&lt;p&gt; LastVote will still have to wait for stepdown to finish, I think, to get the RSTL lock.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;That&apos;s fine, waiting a bit is strictly better than voting no when a node doesn&apos;t need to.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;making lastVote use the JournalFlusher that stepdown interrupts properly.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;If it&apos;s interrupted, then we&apos;d still have the original problem where nodes vote no when they shouldn&apos;t have to, right? Or would that not be interrupted on stepdown?&lt;/p&gt;</comment>
                            <comment id="3081703" author="dianna.hohensee" created="Tue, 12 May 2020 19:13:12 +0000"  >&lt;p&gt;I believe so. Without the mutex, lastVote can&apos;t hold something that stepdown needs in order to proceed. LastVote will still have to wait for stepdown to finish, I think, to get the RSTL lock.&lt;/p&gt;

&lt;p&gt;Lingzhi found a BF where the oplogTruncateAfterPoint is magically set after stepdown clears it, probably by a concurrent thread doing waitUntilDurable during stepdown like lastVote, so I&apos;ve also filed &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-48149&quot; title=&quot;Move callers of waitUntilDurable onto JournalFlusher::waitForJournalFlush&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-48149&quot;&gt;&lt;del&gt;SERVER-48149&lt;/del&gt;&lt;/a&gt; to stop that kind of nonsense. Also a fix for this, now that I think about it, making lastVote use the JournalFlusher that stepdown interrupts properly.&lt;/p&gt;</comment>
                            <comment id="3081564" author="judah.schvimer" created="Tue, 12 May 2020 18:12:31 +0000"  >&lt;p&gt;Great! So, to ensure I understand correctly, if &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-48144&quot; title=&quot;waitUntilDurable should not take a mutex before taking locks&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-48144&quot;&gt;&lt;del&gt;SERVER-48144&lt;/del&gt;&lt;/a&gt; goes according to plan, this deadlock can be closed as Gone Away?&lt;/p&gt;</comment>
                            <comment id="3081307" author="dianna.hohensee" created="Tue, 12 May 2020 17:18:52 +0000"  >&lt;p&gt;Okay, I&apos;ve filed &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-48144&quot; title=&quot;waitUntilDurable should not take a mutex before taking locks&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-48144&quot;&gt;&lt;del&gt;SERVER-48144&lt;/del&gt;&lt;/a&gt; because execution should not be taking a mutex before regular locks &amp;#8211; it&apos;s bad practice and leads to deadlocks like this one &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt; --, regardless of UninterruptibleLockGuards.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-48144&quot; title=&quot;waitUntilDurable should not take a mutex before taking locks&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-48144&quot;&gt;&lt;del&gt;SERVER-48144&lt;/del&gt;&lt;/a&gt; is quick in theory if all goes well &amp;#8211; looking at the code it seems very reasonable --, but will need some performance checks; and it&apos;s always possible something could implode when we change something low level in the code.&lt;/p&gt;</comment>
                            <comment id="3081157" author="dianna.hohensee" created="Tue, 12 May 2020 16:18:40 +0000"  >&lt;p&gt;Looking into this, writing down the deadlock for ease of communication:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;LastVote&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    waitUntilDurable&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        _lastSyncMutex&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            getToken&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                refreshIfPrimary&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                    AutoGetCollection (stuck on RSTL IX)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Stepdown (RSTL X)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    waitUntilDurable&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        _lastSyncMutex (stuck on LastVote&apos;s acquisition)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="3080394" author="judah.schvimer" created="Tue, 12 May 2020 11:57:35 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=dianna.hohensee&quot; class=&quot;user-hover&quot; rel=&quot;dianna.hohensee&quot;&gt;dianna.hohensee&lt;/a&gt;, do you have any ideas on how to safely break this cycle while keeping the LastVote waitUntilDurable call uninterruptible? If not, then we should just revert the C++ changes in the patch and make the elections more robust with retries in &lt;tt&gt;remove_newly_added_field_after_finishing_initial_sync.js&lt;/tt&gt;.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="1348431">SERVER-48144</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10520">
                    <name>Problem/Incident</name>
                                                                <inwardlinks description="is caused by">
                                        <issuelink>
            <issuekey id="1318428">SERVER-47612</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25128"><![CDATA[Replication]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 12 May 2020 11:57:35 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 38 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-1096</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>alexander.golin@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 38 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>25.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-server-repl</customfieldvalue>
            <customfieldvalue>dianna.hohensee@mongodb.com</customfieldvalue>
            <customfieldvalue>judah.schvimer@mongodb.com</customfieldvalue>
            <customfieldvalue>vesselina.ratcheva@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxks13:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hx8233:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxkeaf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>