<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:55:44 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-40700] Deadlock between read prepare conflicts and state transitions</title>
                <link>https://jira.mongodb.org/browse/SERVER-40700</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We do not kill user or internal reads on step down, similarly to how we do not kill internal writes on step down. Thus if they hit a prepare conflict they will cause the deadlock described in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40594&quot; title=&quot;Range deleter in prepare conflict retry loop blocks step down&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40594&quot;&gt;&lt;del&gt;SERVER-40594&lt;/del&gt;&lt;/a&gt;. &lt;/p&gt;

&lt;p&gt;Solution:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Stepdown and step up will now require RSTL in mode S.&lt;/li&gt;
	&lt;li&gt;Rollback requires RSTL in mode X.&lt;/li&gt;
	&lt;li&gt;Reads take RSTL in mode IS.&lt;/li&gt;
	&lt;li&gt;Writes take RSTL in mode IX.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Details:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Operations would need to commit to whether they can write when they acquire a lock, but this is acceptable (and is essentially already the contract).&lt;/li&gt;
	&lt;li&gt;We don&#8217;t plan to implement upgrading locks. If something wants to &quot;upgrade&quot; its locks, it must drop all of its locks and reacquire them and make sure that&apos;s safe for its own purposes.&lt;/li&gt;
	&lt;li&gt;Implement this by taking the same lock mode for the RSTL that you take for the global lock.&lt;/li&gt;
	&lt;li&gt;Advantage that stepdown doesn&#8217;t need to wait for reads to complete/yield and yielded readers don&#8217;t need to wait for stepdown to complete.&lt;/li&gt;
	&lt;li&gt;This fixes the problem for any operation that acquires a global IS lock (user or internal) since stepdown will no longer block on the operation to complete.&lt;/li&gt;
	&lt;li&gt;User operations that acquire global S, IX, or X locks are already killed on stepdown so aren&apos;t a problem.&lt;/li&gt;
	&lt;li&gt;Internal operations that acquire global S, IX, or X locks on user data still must be explicitly killed, so the RangeDeleter, TTL, and any other internal writers to user data must still be audited and fixed.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;The S mode acquisition on step up and step down means concurrent state transitions could start happening. To protect against this we will add a new LockManager ResourceMutex that the ReplicationStateTransationLockGuard acquires after acquiring the RSTL, and releases before it. This should be a straightforward way to allow reads and step-ups/step-downs to not conflict (via S and IS locks) but for step-ups and step-downs to conflict even though they take S locks (via the ResourceMutex that does not interact with reads at all). &lt;/p&gt;</description>
                <environment></environment>
        <key id="740577">SERVER-40700</key>
            <summary>Deadlock between read prepare conflicts and state transitions</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="pavithra.vetriselvan@mongodb.com">Pavithra Vetriselvan</assignee>
                                    <reporter username="judah.schvimer@mongodb.com">Judah Schvimer</reporter>
                        <labels>
                            <label>prepare_durability</label>
                            <label>todo_in_code</label>
                    </labels>
                <created>Wed, 17 Apr 2019 20:05:37 +0000</created>
                <updated>Sun, 29 Oct 2023 22:21:54 +0000</updated>
                            <resolved>Thu, 16 May 2019 14:32:05 +0000</resolved>
                                                    <fixVersion>4.1.12</fixVersion>
                                    <component>Replication</component>
                    <component>Storage</component>
                                        <votes>0</votes>
                                    <watches>12</watches>
                                                                                                                <comments>
                            <comment id="2248918" author="xgen-internal-githook" created="Thu, 16 May 2019 15:10:05 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;email&apos;: &apos;benety@mongodb.com&apos;, &apos;name&apos;: &apos;Benety Goh&apos;, &apos;username&apos;: &apos;benety&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40700&quot; title=&quot;Deadlock between read prepare conflicts and state transitions&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40700&quot;&gt;&lt;del&gt;SERVER-40700&lt;/del&gt;&lt;/a&gt; fix lint again&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/a07de3fde0e627e5f9b0d71a922cc84b1fce44ef&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/a07de3fde0e627e5f9b0d71a922cc84b1fce44ef&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2248912" author="xgen-internal-githook" created="Thu, 16 May 2019 15:04:57 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;email&apos;: &apos;benety@mongodb.com&apos;, &apos;name&apos;: &apos;Benety Goh&apos;, &apos;username&apos;: &apos;benety&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40700&quot; title=&quot;Deadlock between read prepare conflicts and state transitions&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40700&quot;&gt;&lt;del&gt;SERVER-40700&lt;/del&gt;&lt;/a&gt; fix lint&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/8763695da8f2f99c122631ca6a874dcaead3971d&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/8763695da8f2f99c122631ca6a874dcaead3971d&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2248740" author="xgen-internal-githook" created="Thu, 16 May 2019 13:28:32 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Pavi Vetriselvan&apos;, &apos;username&apos;: &apos;pvselvan&apos;, &apos;email&apos;: &apos;pvselvan@umich.edu&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40700&quot; title=&quot;Deadlock between read prepare conflicts and state transitions&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40700&quot;&gt;&lt;del&gt;SERVER-40700&lt;/del&gt;&lt;/a&gt; kill read prepare conflicts during step down&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/953c6116138800e0fbed79e7654eda1690d56f71&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/953c6116138800e0fbed79e7654eda1690d56f71&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2237097" author="suganthi.mani" created="Tue, 7 May 2019 15:14:21 +0000"  >&lt;p&gt;We would not implement the new approach described in the &quot;Description&quot; box. Instead, we will be going with the approach where reads using afterClusterTime that encounter prepare conflicts will be killed during step down.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Implementation Detail:&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;Whenever the read thread hits prepare conflict, it waits either on this conditional variable WiredTigerSessionCache::&lt;a href=&quot;https://github.com/mongodb/mongo/blob/3e537f69b4d65d39be235b971e8f08b7dc876bb1/src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp#L316&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;_prepareCommittedOrAbortedCond&lt;/a&gt; or for interrupt.&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;So, at &lt;a href=&quot;https://github.com/mongodb/mongo/blob/3e537f69b4d65d39be235b971e8f08b7dc876bb1/src/mongo/db/storage/wiredtiger/wiredtiger_prepare_conflict.h#L93&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;line 93&lt;/a&gt;, set &lt;b&gt;&#8220;waitsOnPrepareConflict&#8221;&lt;/b&gt; flag to true indicating that the thread is currently blocked by prepared conflict and that flag can be stored as a new &lt;b&gt;decoration on OperationContext&lt;/b&gt;.&lt;/li&gt;
	&lt;li&gt;Resets the flag to false (via makeGuard or OnBlockExit) once &lt;a href=&quot;https://github.com/mongodb/mongo/blob/3e537f69b4d65d39be235b971e8f08b7dc876bb1/src/mongo/db/storage/wiredtiger/wiredtiger_prepare_conflict.h#L94&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;em&gt;waitUntilPreparedUnitOfWorkCommitsOrAborts&lt;/em&gt;&lt;/a&gt; is finished executing.&lt;/li&gt;
	&lt;li&gt;Make step down&#8217;s &lt;a href=&quot;https://github.com/mongodb/mongo/blob/cf322abb3069b05e6ea32afe53a31381f0c0c6df/src/mongo/db/repl/replication_coordinator_impl.cpp#L1850-L1885&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;RstlKillOpthread&lt;/a&gt; to &lt;a href=&quot;https://github.com/mongodb/mongo/blob/cf322abb3069b05e6ea32afe53a31381f0c0c6df/src/mongo/db/repl/replication_coordinator_impl.cpp#L1834-L1835&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;kill all user operations &lt;/a&gt;&#160;that have flag &#8220;waitsOnPrepareConflict&#8221; set to true.&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="2235428" author="judah.schvimer" created="Mon, 6 May 2019 15:49:16 +0000"  >&lt;p&gt;One note here, after talking with &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=geert.bosch&quot; class=&quot;user-hover&quot; rel=&quot;geert.bosch&quot;&gt;geert.bosch&lt;/a&gt;, is that we can upgrade locks already through lock conversion. It&apos;s possible we don&apos;t currently do this anywhere, but the lock manager supports converting locks from a weaker mode to a stronger mode. This would cause problems with the original design here, but may not with the new plan.&lt;/p&gt;</comment>
                            <comment id="2217605" author="judah.schvimer" created="Thu, 18 Apr 2019 12:10:51 +0000"  >&lt;blockquote&gt;
&lt;p&gt;1) With the new approach, both step down and step up will acquire RSTL in S mode? If yes, Could you update the the sentence &quot;Stepdown will now require RSTL in S.&quot; in the description.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Yes, done.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;This means, read thread(1) and step down thread(2) doesn&apos;t conflict each other. So, I feel we don&apos;t want to kill the operations that acquire global lock in S mode. &lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Great point! I agree. I don&apos;t think we need to kill operations that acquire the global lock in S mode anymore. Can you clarify why the 3 way deadlock only manifested with a global lock in S mode and not with a collection or DB lock in S mode? It seems to me that the following would be possible:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;1. Read acquired RSTL in IX and attempts to acquire collection lock in S mode but blocks behind a prepared transaction holding it.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2. Stepdown blocks on the read to acquire RSTL in mode X.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;3. The prepared transaction blocks on the step down to commit or abort.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;I think this scenario goes away when the read acquires RSTL in mode S or IS and the stepdown acquires the RSTL in mode S so they do not conflict.&lt;/p&gt;

&lt;p&gt;Can you also please clarify why we didn&apos;t need to kill S lock holders on step up? I don&apos;t think we need to anymore with this new design, but I want to understand why we didn&apos;t in the first place.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;We can do a single pass by just keeping track of RSTL lock mode in the locker instead of global lock mode. &lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Can you please remind me why we had the looping behavior in the first place? I looked in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-38511&quot; title=&quot;Implement new stepdown sequence, gated by &#8220;closeConnectionsOnStepdown&#8221;.&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-38511&quot;&gt;&lt;del&gt;SERVER-38511&lt;/del&gt;&lt;/a&gt; but cannot find the motivation.&lt;/p&gt;</comment>
                            <comment id="2217414" author="suganthi.mani" created="Thu, 18 Apr 2019 04:09:26 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=judah.schvimer&quot; class=&quot;user-hover&quot; rel=&quot;judah.schvimer&quot;&gt;judah.schvimer&lt;/a&gt;, I am trying to understand the solution and details in the description. And, I have few initial questions regarding that&lt;/p&gt;

&lt;p&gt;1) With the new approach, both step down and step up will acquire RSTL in S mode? If yes, Could you update the the sentence &lt;b&gt;&quot;Stepdown will now require RSTL in S.&quot;&lt;/b&gt; in the description.&lt;br/&gt;
 2) Currently we are killing global lock in S mode to avoid the below 3 way deadlock&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;1. Read(DBhash) thread acquired  RSTL lock in IX mode , global lock in S mode and waits for collection/DB lock held by prepared transaction.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2. Step down Enqueued RSTL lock in X mode. And, blocked behind read thread.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;3. Prepared transaction waiting for RSTL lock to acquire in IX mode and blocked behind step down thread.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;With the new approach, I assume that the thread that takes global lock in S mode would be considered as reads and they would take RSTL in S mode.This means,&#160;read thread(1) and step down thread(2) doesn&apos;t conflict each other. So, I feel we don&apos;t want to kill the operations that acquire global lock in S mode. &lt;br/&gt;
 3) With the new approach, I also feel the&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/79d96979125e9aaa53774a4e1767b1eb231031a9/src/mongo/db/repl/replication_coordinator_impl.cpp#L1843-L1870&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&#160;looping design&lt;/a&gt;&#160;during step down can be avoided as reads take RSTL in IS and write takes RSTL in IX (equivalent to global lock modes). We can do a single pass by just keeping track of RSTL lock mode in the locker instead of global lock mode.&#160;&lt;/p&gt;

&lt;p&gt;CC &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tess.avitabile&quot; class=&quot;user-hover&quot; rel=&quot;tess.avitabile&quot;&gt;tess.avitabile&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2217154" author="judah.schvimer" created="Wed, 17 Apr 2019 21:15:24 +0000"  >&lt;p&gt;Note that step-up doesn&apos;t need to kill operations since the only operations that can take place outside of secondary batch application on secondaries are reads that will no longer conflict with the RSTL. We currently kill operations that acquire global locks in mode S, but we may be able to relax that check here. Otherwise we may need to kill those operations on step-up.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                        <issuelink>
            <issuekey id="759654">SERVER-41037</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="760760">SERVER-41057</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="759571">SERVER-41033</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="759582">SERVER-41034</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="759612">SERVER-41035</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="759619">SERVER-41036</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="881085">SERVER-42537</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="736582">SERVER-40594</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="736492">SERVER-40586</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="738609">SERVER-40641</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="630322">SERVER-37988</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="730211">SERVER-40487</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 18 Apr 2019 04:09:26 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 38 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-1032</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 38 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>judah.schvimer@mongodb.com</customfieldvalue>
            <customfieldvalue>pavithra.vetriselvan@mongodb.com</customfieldvalue>
            <customfieldvalue>suganthi.mani@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|huuuin:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hukaen:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2919">Repl 2019-05-06</customfieldvalue>
    <customfieldvalue id="2920">Repl 2019-05-20</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|huugrz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>