<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:06:14 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-44529] Re-acquiring locks after a yield and step down results in wrong parameters on recoveryUnit</title>
                <link>https://jira.mongodb.org/browse/SERVER-44529</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;When we acquire collection locks with &lt;tt&gt;AutoGetCollectionForRead&lt;/tt&gt; on a primary, we by default set the &lt;tt&gt;TimestampReadSource&lt;/tt&gt;  on the recovery unit to &lt;tt&gt;kUnset&lt;/tt&gt;.  However, when we re-acquire locks after a yield in &lt;tt&gt;PlanExecutor&lt;/tt&gt;, we still use that same &lt;tt&gt;TimestampReadSource&lt;/tt&gt; even if we have stepped down. Since reads survive replication state changes, that can result in us reading from within the middle of a replication batch, which means reading possibly causally-related data out of order.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Note:&lt;/b&gt; This ticket initially described both step up and step down, but step up is now separated out into &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-46721&quot; title=&quot;Step up may cause reads at PIT with holes after yielding&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-46721&quot;&gt;&lt;del&gt;SERVER-46721&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Potential fix:&lt;/b&gt; After recovery from yield, change the read source to &lt;tt&gt;kNoOverlap&lt;/tt&gt; if it was &lt;tt&gt;kUnset&lt;/tt&gt; or &lt;tt&gt;kNoTimestamp&lt;/tt&gt;.&lt;/p&gt;</description>
                <environment></environment>
        <key id="994549">SERVER-44529</key>
            <summary>Re-acquiring locks after a yield and step down results in wrong parameters on recoveryUnit</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="louis.williams@mongodb.com">Louis Williams</assignee>
                                    <reporter username="matthew.russotto@mongodb.com">Matthew Russotto</reporter>
                        <labels>
                            <label>bkp</label>
                    </labels>
                <created>Fri, 8 Nov 2019 21:24:58 +0000</created>
                <updated>Sun, 29 Oct 2023 22:15:08 +0000</updated>
                            <resolved>Fri, 15 May 2020 15:09:10 +0000</resolved>
                                                    <fixVersion>4.4.0-rc7</fixVersion>
                    <fixVersion>4.2.9</fixVersion>
                    <fixVersion>4.7.0</fixVersion>
                                    <component>Querying</component>
                    <component>Replication</component>
                    <component>Storage</component>
                                        <votes>0</votes>
                                    <watches>18</watches>
                                                                                                                <comments>
                            <comment id="3324807" author="xgen-internal-githook" created="Thu, 6 Aug 2020 16:07:04 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Louis Williams&apos;, &apos;email&apos;: &apos;louis.williams@mongodb.com&apos;, &apos;username&apos;: &apos;louiswilliams&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-46721&quot; title=&quot;Step up may cause reads at PIT with holes after yielding&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-46721&quot;&gt;&lt;del&gt;SERVER-46721&lt;/del&gt;&lt;/a&gt; Secondary readers should read at the no-overlap time instead of lastApplied&lt;/p&gt;

&lt;p&gt;The no-overlap time, ReadSource::kNoOverlap, is the minimum of replication&apos;s lastApplied timestamp&lt;br/&gt;
and WiredTiger&apos;s all_durable time. This time is independent of replication state and ensures&lt;br/&gt;
queries do not see oplog holes after state transitions from secondary to primary.&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 25c694f365db0f07a445bd17b6cd5cbf32f5f2f9)&lt;br/&gt;
(cherry picked from commit 5ddf9db5635e4af2d1295bd2f1007e86d517c6c2)&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-44529&quot; title=&quot;Re-acquiring locks after a yield and step down results in wrong parameters on recoveryUnit&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-44529&quot;&gt;&lt;del&gt;SERVER-44529&lt;/del&gt;&lt;/a&gt; Query yield recovery after a stepdown should switch to read at the no-overlap time&lt;/p&gt;

&lt;p&gt;After yielding and reacquiring locks in a query, the preconditions that were used to select our&lt;br/&gt;
ReadSource initially need to be checked again. Queries hold an AutoGetCollectionForRead RAII&lt;br/&gt;
lock for their lifetime, which may select a ReadSource based on state (e.g. replication&lt;br/&gt;
state). After a query yields its locks, this state may have changed, invalidating our current&lt;br/&gt;
choice of ReadSource.&lt;/p&gt;

&lt;p&gt;(cherry picked from commit b3a5b5258a487006f0487b9e7e0a1d0d4f1119ff)&lt;br/&gt;
(cherry picked from commit ce57ddce19e642401ec1b56871b3a9402116ccfc)&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-48475&quot; title=&quot;Secondary reads always calculate all_durable despite only using lastApplied&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-48475&quot;&gt;&lt;del&gt;SERVER-48475&lt;/del&gt;&lt;/a&gt; Reimplement lastApplied for secondary reads&lt;/p&gt;

&lt;p&gt;This partially reverts work to use the kNoOverlap ReadSource on secondaries since the all_durable&lt;br/&gt;
calculation is unnecessary and expensive.&lt;/p&gt;

&lt;p&gt;(cherry picked from commit ff92d4435fa75c2b947c49cffaf48805b320a5ae)&lt;br/&gt;
(cherry picked from commit 4e3f3b96826772baa777b7563b38574c1e11c2e4)&lt;br/&gt;
Branch: v4.2&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/caef6619556ddeb20adf8e2bad651b96d2b3ea70&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/caef6619556ddeb20adf8e2bad651b96d2b3ea70&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3196979" author="louis.williams" created="Tue, 9 Jun 2020 13:29:30 +0000"  >&lt;p&gt;Due to a small performance regression, I have partially reverted this work in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-48475&quot; title=&quot;Secondary reads always calculate all_durable despite only using lastApplied&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-48475&quot;&gt;&lt;del&gt;SERVER-48475&lt;/del&gt;&lt;/a&gt;. Secondary reads no longer use kNoOverlap, but use kLastApplied as they did before &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-46721&quot; title=&quot;Step up may cause reads at PIT with holes after yielding&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-46721&quot;&gt;&lt;del&gt;SERVER-46721&lt;/del&gt;&lt;/a&gt;. Since the no-overlap point on secondaries is always equal to lastApplied, we were performing unnecessary all_durable calculations for every read.&lt;/p&gt;</comment>
                            <comment id="3095573" author="xgen-internal-githook" created="Tue, 19 May 2020 17:16:24 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Louis Williams&apos;, &apos;email&apos;: &apos;louis.williams@mongodb.com&apos;, &apos;username&apos;: &apos;louiswilliams&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-44529&quot; title=&quot;Re-acquiring locks after a yield and step down results in wrong parameters on recoveryUnit&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-44529&quot;&gt;&lt;del&gt;SERVER-44529&lt;/del&gt;&lt;/a&gt; Query yield recovery after a stepdown should switch to read at the no-overlap time&lt;/p&gt;

&lt;p&gt;After yielding and reacquiring locks in a query, the preconditions that were used to select our&lt;br/&gt;
ReadSource initially need to be checked again. Queries hold an AutoGetCollectionForRead RAII&lt;br/&gt;
lock for their lifetime, which may select a ReadSource based on state (e.g. replication&lt;br/&gt;
state). After a query yields its locks, this state may have changed, invalidating our current&lt;br/&gt;
choice of ReadSource.&lt;/p&gt;

&lt;p&gt;(cherry picked from commit b3a5b5258a487006f0487b9e7e0a1d0d4f1119ff)&lt;br/&gt;
Branch: v4.4&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/ce57ddce19e642401ec1b56871b3a9402116ccfc&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/ce57ddce19e642401ec1b56871b3a9402116ccfc&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3089881" author="xgen-internal-githook" created="Fri, 15 May 2020 14:56:08 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Louis Williams&apos;, &apos;email&apos;: &apos;louis.williams@mongodb.com&apos;, &apos;username&apos;: &apos;louiswilliams&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-44529&quot; title=&quot;Re-acquiring locks after a yield and step down results in wrong parameters on recoveryUnit&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-44529&quot;&gt;&lt;del&gt;SERVER-44529&lt;/del&gt;&lt;/a&gt; Query yield recovery after a stepdown should switch to read at the no-overlap time&lt;/p&gt;

&lt;p&gt;After yielding and reacquiring locks in a query, the preconditions that were used to select our&lt;br/&gt;
ReadSource initially need to be checked again. Queries hold an AutoGetCollectionForRead RAII&lt;br/&gt;
lock for their lifetime, which may select a ReadSource based on state (e.g. replication&lt;br/&gt;
state). After a query yields its locks, this state may have changed, invalidating our current&lt;br/&gt;
choice of ReadSource.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/b3a5b5258a487006f0487b9e7e0a1d0d4f1119ff&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/b3a5b5258a487006f0487b9e7e0a1d0d4f1119ff&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3058640" author="louis.williams" created="Tue, 28 Apr 2020 14:57:08 +0000"  >&lt;p&gt;Before &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-38511&quot; title=&quot;Implement new stepdown sequence, gated by &#8220;closeConnectionsOnStepdown&#8221;.&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-38511&quot;&gt;&lt;del&gt;SERVER-38511&lt;/del&gt;&lt;/a&gt;, reads did not survive the transition from primary to secondary, so I am declining the 4.0 backport.&lt;/p&gt;</comment>
                            <comment id="3057379" author="louis.williams" created="Mon, 27 Apr 2020 20:01:03 +0000"  >&lt;p&gt;I&apos;ve spent some time rationalizing why the proposed solution is correct. The main problem that we want to avoid is allowing an operation to read without a timestamp and then switch to reading with a timestamp. Unaddressed, this visibility problem will cause will once-visible writes to vanish (see &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-41361&quot; title=&quot;Do not read at lastApplied while already holding the PBWM lock on secondaries&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-41361&quot;&gt;&lt;del&gt;SERVER-41361&lt;/del&gt;&lt;/a&gt;), and can lead to crashes.&lt;/p&gt;

&lt;p&gt;The proposed solution in the event of a stepdown after recovering from a yield is the following: if we were reading without a timestamp before stepdown, read at &lt;a href=&quot;https://github.com/mongodb/mongo/blob/53fbcd81aa71a5d5289fdc27dffe47f174039e5e/src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.cpp#L615&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;kNoOverlap&lt;/a&gt;, which is the minimum of &lt;tt&gt;lastApplied&lt;/tt&gt; and &lt;tt&gt;all_durable,&lt;/tt&gt;&#160;falling-back to the solution in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-46721&quot; title=&quot;Step up may cause reads at PIT with holes after yielding&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-46721&quot;&gt;&lt;del&gt;SERVER-46721&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;This is a scenario I believe demonstrates that it is not possible for a snapshot to &quot;backtrack&quot;.&#160; For example, say the &lt;tt&gt;all_durable&lt;/tt&gt; is T1 because of a hole at T2. A client reads without a timestamp, seeing writes at T3, which is equal to &lt;tt&gt;lastApplied&lt;/tt&gt;. If a stepdown occurs, all holes are closed because active transactions will either commit just-in-time or be aborted by stepdown. This means that immediately after stepdown, &lt;tt&gt;all_durable&lt;/tt&gt; becomes equal to &lt;tt&gt;lastApplied&lt;/tt&gt;, or T3 (assuming no other writes). More importantly, if we started a read without a timestamp just before stepdown, after stepdown it would be impossible for &lt;tt&gt;kNoOverlap&lt;/tt&gt; (the minimum of &lt;tt&gt;all_durable&lt;/tt&gt; and &lt;tt&gt;lastApplied&lt;/tt&gt;) to move backward and read at a timestamp before T3. This should provide an example of why this solution is safe for replicated collections.&lt;/p&gt;

&lt;p&gt;&lt;del&gt;For the oplog: reading without a timestamp is always safe, regardless of the replication state. We track an &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a30cef00b9e3ccfe60a4dce7f9f2c7cbaf1e68b7/src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.cpp#L506&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;oplog visibility, &quot;no holes&quot;, timestamp&lt;/a&gt;. Our solution should take care to ensure the oplog opts-out of this new behavior.&lt;/del&gt;&lt;/p&gt;

&lt;p&gt;Update: my last statement about reading from the oplog on secondaries being safe is not confirmed&lt;/p&gt;

&lt;p&gt;Update: we need a different solution on 4.0 where the kNoOverlap ReadSource doesn&apos;t exist. We will probably need to partially backport&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-38986&quot; title=&quot;Timestamp drained writes from the index build side table to the index &quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-38986&quot;&gt;&lt;del&gt;SERVER-38986&lt;/del&gt;&lt;/a&gt;, where it was introduced.&lt;/p&gt;</comment>
                            <comment id="2913127" author="matthew.russotto" created="Wed, 26 Feb 2020 21:40:20 +0000"  >&lt;p&gt;It looks like this bug exists on 4.0, but the 3.6 AutoGetCollectionForRead is completely different so it shouldn&apos;t be affected.&lt;/p&gt;</comment>
                            <comment id="2912594" author="tess.avitabile" created="Wed, 26 Feb 2020 18:13:04 +0000"  >&lt;p&gt;Does this need to be backported further back than 4.2? I think it&apos;s the case on earlier versions that reads survive the transition from secondary to primary.&lt;/p&gt;</comment>
                            <comment id="2528943" author="steven.vannelli" created="Mon, 11 Nov 2019 18:09:20 +0000"  >&lt;p&gt;Assigning this to the Execution. Please reach out to Replication team if support is needed.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="1253817">SERVER-46721</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="1416196">SERVER-49781</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="1289074">SERVER-47084</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1365768">SERVER-48475</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>12.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 11 Nov 2019 18:09:20 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 26 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/SERVER-46721'>SERVER-46721</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 26 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>13.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>louis.williams@mongodb.com</customfieldvalue>
            <customfieldvalue>matthew.russotto@mongodb.com</customfieldvalue>
            <customfieldvalue>steven.vannelli@mongodb.com</customfieldvalue>
            <customfieldvalue>tess.avitabile@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hw1dfr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr5gvr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3400">Execution Team 2019-12-02</customfieldvalue>
    <customfieldvalue id="3401">Execution Team 2019-12-16</customfieldvalue>
    <customfieldvalue id="3517">Execution Team 2020-01-13</customfieldvalue>
    <customfieldvalue id="3518">Execution Team 2020-01-27</customfieldvalue>
    <customfieldvalue id="3519">Execution Team 2020-02-10</customfieldvalue>
    <customfieldvalue id="3521">Execution Team 2019-12-30</customfieldvalue>
    <customfieldvalue id="3630">Execution Team 2020-02-24</customfieldvalue>
    <customfieldvalue id="3631">Execution Team 2020-03-09</customfieldvalue>
    <customfieldvalue id="3632">Execution Team 2020-03-23</customfieldvalue>
    <customfieldvalue id="3633">Execution Team 2020-04-06</customfieldvalue>
    <customfieldvalue id="3635">Execution Team 2020-05-04</customfieldvalue>
    <customfieldvalue id="3876">Execution Team 2020-05-18</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hw0zp3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>