<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:51:12 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-60969] ReplClientInfo::getLastOp may not be updated before being used to wait for replication in tenant migration</title>
                <link>https://jira.mongodb.org/browse/SERVER-60969</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;&lt;tt&gt;waitUntilMigrationReachesReturnAfterReachingTimestamp&#160;&lt;/tt&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/598b99c1be1aa389407f25ea773b2e95141e3dd4/src/mongo/db/repl/tenant_migration_recipient_service.cpp#L518&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;updates the &lt;tt&gt;rejectReadsBeforeTimestamp&lt;/tt&gt;&lt;/a&gt;&#160;in the recipient state doc, and then use &lt;a href=&quot;https://github.com/mongodb/mongo/blob/598b99c1be1aa389407f25ea773b2e95141e3dd4/src/mongo/db/repl/tenant_migration_recipient_service.cpp#L524&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;ReplClientInfo::getLastOp()&lt;/tt&gt;&lt;/a&gt;&#160; to get the latest opTime written by this client. This opTime is later being used to &lt;a href=&quot;https://github.com/mongodb/mongo/blob/598b99c1be1aa389407f25ea773b2e95141e3dd4/src/mongo/db/repl/tenant_migration_recipient_service.cpp#L529&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;wait for replication&lt;/a&gt;. However in some cases this opTime may not be initialized at all, so the wait for replication will return immediately. This causes the &lt;tt&gt;recipientSyncData&lt;/tt&gt; command to return without replicating the state doc updates to majority nodes, which is not correct because the donor might incorrectly think the migration is finished.&lt;/p&gt;

&lt;p&gt;One scenario for this to happen:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Recipient reaches consistent state&lt;/li&gt;
	&lt;li&gt;Donor sends &lt;tt&gt;recipientSyncData&lt;/tt&gt;&#160;command with&#160;&lt;tt&gt;returnAfterReachingTimestamp&lt;/tt&gt; to recipient&lt;/li&gt;
&lt;/ol&gt;


&lt;ol&gt;
	&lt;li&gt;Donor starts failing over and the new donor primary sends another&#160;&lt;tt&gt;recipientSyncData&lt;/tt&gt;&#160;command with the same&#160;&lt;tt&gt;returnAfterReachingTimestamp&lt;/tt&gt;&lt;/li&gt;
	&lt;li&gt;The first&#160;&lt;tt&gt;recipientSyncData command&lt;/tt&gt;&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/598b99c1be1aa389407f25ea773b2e95141e3dd4/src/mongo/db/repl/tenant_migration_recipient_service.cpp#L518&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;updates the &lt;tt&gt;rejectReadsBeforeTimestamp&lt;/tt&gt;&lt;/a&gt;&#160;in the state doc, but fails to &lt;a href=&quot;https://github.com/mongodb/mongo/blob/598b99c1be1aa389407f25ea773b2e95141e3dd4/src/mongo/db/repl/tenant_migration_recipient_service.cpp#L529&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;wait for replication&lt;/a&gt;&#160;due to being stepped down&lt;/li&gt;
	&lt;li&gt;The second&#160;&lt;tt&gt;recipientSyncData&lt;/tt&gt;&#160;also starts updating the &lt;tt&gt;rejectReadsBeforeTimestamp&lt;/tt&gt;, but since the state doc was already updated in memory with the same&#160;&lt;tt&gt;rejectReadsBeforeTimestamp&lt;/tt&gt;, this will be a no-op and we &lt;a href=&quot;https://github.com/mongodb/mongo/blob/598b99c1be1aa389407f25ea773b2e95141e3dd4/src/mongo/db/op_observer_impl.cpp#L641-L644&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;do not write anything to the oplog&lt;/a&gt;. This means that the &lt;tt&gt;ReplClientInfo::getLastOp()&lt;/tt&gt;&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/598b99c1be1aa389407f25ea773b2e95141e3dd4/src/mongo/db/repl/oplog.cpp#L384&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;will not be updated&lt;/a&gt;. So we end up waiting on an uninitialized opTime since no writes ever happened on this client, and&#160;the wait for replication will return immediately.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;This pattern (getting opTime from&#160;&lt;tt&gt;ReplClientInfo::getLastOp()&lt;/tt&gt; and wait on that opTime) is being used in multiple places in tenant migration, so this issue may happen elsewhere.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1908475">SERVER-60969</key>
            <summary>ReplClientInfo::getLastOp may not be updated before being used to wait for replication in tenant migration</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="jesse@mongodb.com">A. Jesse Jiryu Davis</assignee>
                                    <reporter username="wenbin.zhu@mongodb.com">Wenbin Zhu</reporter>
                        <labels>
                    </labels>
                <created>Mon, 25 Oct 2021 16:14:30 +0000</created>
                <updated>Sun, 29 Oct 2023 21:46:58 +0000</updated>
                            <resolved>Thu, 11 Nov 2021 20:49:34 +0000</resolved>
                                                    <fixVersion>5.2.0</fixVersion>
                    <fixVersion>5.1.1</fixVersion>
                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>6</watches>
                                                                                                                <comments>
                            <comment id="4190852" author="xgen-internal-githook" created="Tue, 16 Nov 2021 01:42:33 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;A. Jesse Jiryu Davis&apos;, &apos;email&apos;: &apos;jesse@mongodb.com&apos;, &apos;username&apos;: &apos;ajdavis&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-60969&quot; title=&quot;ReplClientInfo::getLastOp may not be updated before being used to wait for replication in tenant migration&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-60969&quot;&gt;&lt;del&gt;SERVER-60969&lt;/del&gt;&lt;/a&gt; Race in tenant migration state machine, try 2&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 65b6cd2fc68021388b992fad6d46fa349324f2a2)&lt;br/&gt;
Branch: v5.1&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/71e85a2029d7e93f2915191ea28be10e2d869859&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/71e85a2029d7e93f2915191ea28be10e2d869859&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4182667" author="xgen-internal-githook" created="Thu, 11 Nov 2021 12:28:13 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;A. Jesse Jiryu Davis&apos;, &apos;email&apos;: &apos;jesse@mongodb.com&apos;, &apos;username&apos;: &apos;ajdavis&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-60969&quot; title=&quot;ReplClientInfo::getLastOp may not be updated before being used to wait for replication in tenant migration&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-60969&quot;&gt;&lt;del&gt;SERVER-60969&lt;/del&gt;&lt;/a&gt; Race in tenant migration state machine, try 2&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/65b6cd2fc68021388b992fad6d46fa349324f2a2&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/65b6cd2fc68021388b992fad6d46fa349324f2a2&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4172897" author="xgen-internal-githook" created="Fri, 5 Nov 2021 22:12:37 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;A. Jesse Jiryu Davis&apos;, &apos;email&apos;: &apos;jesse@mongodb.com&apos;, &apos;username&apos;: &apos;ajdavis&apos;}
&lt;p&gt;Message: Revert &quot;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-60969&quot; title=&quot;ReplClientInfo::getLastOp may not be updated before being used to wait for replication in tenant migration&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-60969&quot;&gt;&lt;del&gt;SERVER-60969&lt;/del&gt;&lt;/a&gt; Race in tenant migration state machine&quot;&lt;/p&gt;

&lt;p&gt;This reverts commit 0eda431672cedbbe62e3b487d2389b1f5c5695df.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/cb50e527b2d886d68ab831abe6d8dc61d63237bd&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/cb50e527b2d886d68ab831abe6d8dc61d63237bd&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4170662" author="xgen-internal-githook" created="Fri, 5 Nov 2021 00:03:26 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;A. Jesse Jiryu Davis&apos;, &apos;email&apos;: &apos;jesse@mongodb.com&apos;, &apos;username&apos;: &apos;ajdavis&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-60969&quot; title=&quot;ReplClientInfo::getLastOp may not be updated before being used to wait for replication in tenant migration&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-60969&quot;&gt;&lt;del&gt;SERVER-60969&lt;/del&gt;&lt;/a&gt; Race in tenant migration state machine&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/0eda431672cedbbe62e3b487d2389b1f5c5695df&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/0eda431672cedbbe62e3b487d2389b1f5c5695df&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4168476" author="suganthi.mani" created="Wed, 3 Nov 2021 23:52:28 +0000"  >&lt;p&gt;Here are *&lt;b&gt;few updates&lt;/b&gt;*:&lt;/p&gt;

&lt;p&gt;I was wondering whether the fix needs to be applied to donor side as well. Thinking about it,  made me to doubt the scenario mentioned in the problem description as it describes that the new recipient primary already has the state document w/ RejectReadsBeforeTimestamp set (which is non-majority committed), resulted in no-op write and returning the recipientSyncData command response w/o ensuring the state doc is majority committed and eventually leading to rollback of the  RejectReadsBeforeTimestamp update . &lt;/p&gt;

&lt;p&gt;To my understanding, POS will start the recipient instance using the &lt;a href=&quot;https://github.com/10gen/mongo/blob/2d5a16adde2b17c851675c6a95f84afa33347cd1/src/mongo/db/repl/primary_only_service.cpp#L370-L392&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;majority-committed state document&lt;/a&gt;. So, which means, there is no possibility  that the recipientSyncData command response was returned before the state doc w/ RejectReadsBeforeTimestamp update was majority committed.&lt;/p&gt;

&lt;p&gt;Spoke With &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=wenbin.zhu&quot; class=&quot;user-hover&quot; rel=&quot;wenbin.zhu&quot;&gt;wenbin.zhu&lt;/a&gt; and we realized the scenario that caused BF failure is that  (also aligns with BF-22917 logs)&lt;br/&gt;
1) Donor D1 (primary) sends  recipientSyncData w/ returnAfterReachingDonorTimestamp to Recipient R1 (primary).&lt;br/&gt;
2) Donor D1 steps down.&lt;br/&gt;
3) Donor D2 steps up and becomes a primary;  it also sends recipientSyncData w/ returnAfterReachingDonorTimestamp to Recipient R1.&lt;br/&gt;
4) R1 received 2 concurrent recipientSyncData cmds, one from stale primary D1 and other from active primary D2. &lt;a href=&quot;https://github.com/mongodb/mongo/blob/598b99c1be1aa389407f25ea773b2e95141e3dd4/src/mongo/db/repl/tenant_migration_recipient_service.cpp#L454&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;waitUntilMigrationReachesReturnAfterReachingTimestamp()&lt;/a&gt; is not entirely serialized. Both commands wait for block timestamp to be majority committed and then, try to update the state doc w/ RejectReadsBeforeTimestamp, cmd from stale primary D1 updated the document.  So, cmd from active primary D2  did a no-op.&lt;br/&gt;
&lt;b&gt;5)As a result, cmd from active primary D2 didn&apos;t wait for the state doc to be majority replicated.&lt;/b&gt; and returned response to active primary D1&lt;br/&gt;
6) Now, cmd from stale primary D1 waits for the state doc to be majority committed, but R1 steps down and eventually lead to rollback of the  RejectReadsBeforeTimestamp update&lt;/p&gt;

&lt;p&gt;&lt;b&gt;The problem is that &lt;a href=&quot;https://github.com/mongodb/mongo/blob/598b99c1be1aa389407f25ea773b2e95141e3dd4/src/mongo/db/repl/tenant_migration_recipient_service.cpp#L454&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;waitUntilMigrationReachesReturnAfterReachingTimestamp()&lt;/a&gt; which updates state document,  is not concurrent safe.&lt;/b&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;This pattern (getting opTime from ReplClientInfo::getLastOp() and wait on that opTime) is being used in multiple places in tenant migration, so this issue may happen elsewhere.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;Having said that, this pattern is currently safer in all places of tenant migration code (both donor &amp;amp; recipient) except &lt;a href=&quot;https://github.com/mongodb/mongo/blob/598b99c1be1aa389407f25ea773b2e95141e3dd4/src/mongo/db/repl/tenant_migration_recipient_service.cpp#L454&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this&lt;/a&gt; place where we update the state doc for &quot;RejectReadsBeforeTimestamp&quot; because that&apos;s the only place we update the state doc outside the future chain. I believe, we initially wrote the tenant migration code with the assumption that the state doc gets modified only via the chain. The set &quot;RejectReadsBeforeTimestamp&quot; seems to be exceptional where the state doc gets updated in the command caller path.&lt;/p&gt;</comment>
                            <comment id="4146799" author="lingzhi.deng" created="Mon, 25 Oct 2021 21:44:27 +0000"  >&lt;p&gt;FWIW, client writes use the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/16d4013b9dc4ad4814a1b586278ebb7e95e340c6/src/mongo/db/ops/write_ops_exec.cpp#L177&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;LastOpFixer&lt;/tt&gt;&lt;/a&gt; to handle noop writes. We may want to consider using that for internal writes in tenant migration.&lt;/p&gt;</comment>
                            <comment id="4145803" author="JIRAUSER1258790" created="Mon, 25 Oct 2021 16:22:57 +0000"  >&lt;p&gt;cc &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=lingzhi.deng&quot; class=&quot;user-hover&quot; rel=&quot;lingzhi.deng&quot;&gt;lingzhi.deng&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4145801" author="JIRAUSER1258790" created="Mon, 25 Oct 2021 16:22:41 +0000"  >&lt;p&gt;I did some hack to quickly repro this issue by &lt;a href=&quot;https://github.com/mongodb/mongo/blob/598b99c1be1aa389407f25ea773b2e95141e3dd4/src/mongo/db/repl/tenant_migration_recipient_service.cpp#L518&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;setting the rejectReadsBeforeTimestamp&lt;/a&gt; directly to the &lt;tt&gt;returnAfterReachingTimestamp&lt;/tt&gt; instead of setting to the result of &lt;tt&gt;selectRejectReadsBeforeTimestamp,&lt;/tt&gt;&#160;which returns the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/598b99c1be1aa389407f25ea773b2e95141e3dd4/src/mongo/db/repl/tenant_migration_recipient_service.cpp#L450&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;maximum of&#160;finalRecipientWriteTimestamp and returnAfterReachingTimestamp.&lt;/a&gt;&#160;In reality &lt;tt&gt;returnAfterReachingTimestamp&lt;/tt&gt;&#160;could be greater than &lt;tt&gt;finalRecipientWriteTimestamp&lt;/tt&gt;, so the repro should be valid.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10520">
                    <name>Problem/Incident</name>
                                            <outwardlinks description="causes">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="1923013">SERVER-61404</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="22495"><![CDATA[v5.1]]></customfieldvalue>
    <customfieldvalue key="21777"><![CDATA[v5.0]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 25 Oct 2021 21:44:27 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 12 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 12 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>167.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>jesse@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>lingzhi.deng@mongodb.com</customfieldvalue>
            <customfieldvalue>suganthi.mani@mongodb.com</customfieldvalue>
            <customfieldvalue>wenbin.zhu@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i07ge7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hzr167:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="5546">Server Serverless 2021-11-01</customfieldvalue>
    <customfieldvalue id="5553">Server Serverless 2021-11-15</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i072jj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>