<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:51:49 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-39364] Audit uses of setLastOpToSystemLastOpTime</title>
                <link>https://jira.mongodb.org/browse/SERVER-39364</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;It is not valid to call &lt;tt&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/25b7af8b7367de11f0d4d864bd6a51983227c494/src/mongo/db/repl/repl_client_info.h#L72&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;setLastOpToSystemLastOpTime()&lt;/a&gt;&lt;/tt&gt; and then wait for majority writeConcern to ensure that any data read was majority committed.&#160;The reason is that writes update the lastApplied in an on-commit hook after the data changes are made. Consider the following scenario. Originally the lastApplied is time 1. Operation A updates the document {&lt;tt&gt;_id: 0&lt;/tt&gt;} to have a=5 and writes an oplog entry with time 2. Operation B reads the document {&lt;tt&gt;_id: 0, a: 5&lt;/tt&gt;}.&#160; Operation B calls &lt;tt&gt;setLastOpToSystemLastOpTime()&lt;/tt&gt; and&#160;gets the lastApplied time as time 1. Then the on-commit hook for Operation A updates the lastApplied to time 2. Operation B will wait for the majority commit point to reach time 1, which does not guarantee that the data that it read was majority committed.&lt;/p&gt;

&lt;p&gt;Any callers of &lt;tt&gt;setLastOpToSystemLastOpTime()&lt;/tt&gt; that use it to ensure data read was majority committed are suspect. This excludes reading transaction state in transaction operations, since they are serialized by the session checkout mechanism, so a new transaction operation cannot start until the on-commit hooks have finished for the previous operation.&lt;/p&gt;</description>
                <environment></environment>
        <key id="683182">SERVER-39364</key>
            <summary>Audit uses of setLastOpToSystemLastOpTime</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="lingzhi.deng@mongodb.com">Lingzhi Deng</assignee>
                                    <reporter username="tess.avitabile@mongodb.com">Tess Avitabile</reporter>
                        <labels>
                    </labels>
                <created>Mon, 4 Feb 2019 15:09:23 +0000</created>
                <updated>Sun, 29 Oct 2023 22:24:27 +0000</updated>
                            <resolved>Wed, 15 Jan 2020 17:14:03 +0000</resolved>
                                                    <fixVersion>4.3.3</fixVersion>
                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>8</watches>
                                                                                                                <comments>
                            <comment id="2731896" author="xgen-internal-githook" created="Wed, 15 Jan 2020 16:18:53 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Lingzhi Deng&apos;, &apos;email&apos;: &apos;lingzhi.deng@mongodb.com&apos;, &apos;username&apos;: &apos;ldennis&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-39364&quot; title=&quot;Audit uses of setLastOpToSystemLastOpTime&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-39364&quot;&gt;&lt;del&gt;SERVER-39364&lt;/del&gt;&lt;/a&gt;: Get the last oplog entry from the storage engine for setLastOpToSystemLastOpTime&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/756a87f2e86d8f67259b5995d8f1cf7dcc27f7a6&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/756a87f2e86d8f67259b5995d8f1cf7dcc27f7a6&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2636886" author="tess.avitabile" created="Tue, 17 Dec 2019 16:20:53 +0000"  >&lt;p&gt;Your plan sounds great. Thanks for the thorough investigation. For transactions, you can repurpose&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-41165&quot; title=&quot;Transaction participants don&amp;#39;t need to do a noop write before committing a read-only transaction&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-41165&quot;&gt;SERVER-41165&lt;/a&gt; to remove the noop write. I agree that the solution for change streams using timestamped reads does not need to change, and that the performance of linearizable is lower priority.&lt;/p&gt;</comment>
                            <comment id="2635300" author="judah.schvimer" created="Mon, 16 Dec 2019 22:16:18 +0000"  >&lt;p&gt;sgtm&lt;/p&gt;</comment>
                            <comment id="2635198" author="lingzhi.deng" created="Mon, 16 Dec 2019 21:45:32 +0000"  >&lt;p&gt;Talked with &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=judah.schvimer&quot; class=&quot;user-hover&quot; rel=&quot;judah.schvimer&quot;&gt;judah.schvimer&lt;/a&gt; offline, in this ticket, we will re-implement &lt;tt&gt;ReplClientInfo::setLastOpToSystemLastOpTime&lt;/tt&gt; to use &lt;a href=&quot;https://github.com/mongodb/mongo/blob/bc754e818ce974352bd76c850ec8dbc36620e765/src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp#L1373&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;WiredTigerRecordStore::getLatestOplogTimestamp&lt;/tt&gt;&lt;/a&gt; implemented in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-43656&quot; title=&quot;Return the timestamp of the top of the oplog in the storage integration layer&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-43656&quot;&gt;&lt;del&gt;SERVER-43656&lt;/del&gt;&lt;/a&gt;. To address &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jason.chan&quot; class=&quot;user-hover&quot; rel=&quot;jason.chan&quot;&gt;jason.chan&lt;/a&gt; comment, we will only grab global MODE_IS lock when accessing the oplog collection and the record store. We can do that via &lt;tt&gt;LocalOplogInfo&lt;/tt&gt;.&lt;/p&gt;

&lt;p&gt;And once the (re)implementation is done, I will audit callers of &lt;tt&gt;setLastOpToSystemLastOpTime&lt;/tt&gt; and file tickets to remove unnecessary noop oplog writes that we did in the past for correctness in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-38906&quot; title=&quot;Multi-document transactions should not perform timestamped read ahead of all-committed&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-38906&quot;&gt;&lt;del&gt;SERVER-38906&lt;/del&gt;&lt;/a&gt;. &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-39356&quot; title=&quot;Change stream updateLookup queries with speculative majority may return uncommitted data&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-39356&quot;&gt;&lt;del&gt;SERVER-39356&lt;/del&gt;&lt;/a&gt; was done using timestamped reads for change streams. So I think we could leave that as is. And &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-37948&quot; title=&quot;Linearizable read concern is not satisfied by getMores on a cursor&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-37948&quot;&gt;&lt;del&gt;SERVER-37948&lt;/del&gt;&lt;/a&gt; is only for linearizable and it is probably less value. So we could leave that too?&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tess.avitabile&quot; class=&quot;user-hover&quot; rel=&quot;tess.avitabile&quot;&gt;tess.avitabile&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=judah.schvimer&quot; class=&quot;user-hover&quot; rel=&quot;judah.schvimer&quot;&gt;judah.schvimer&lt;/a&gt; How does the plan sound to you? Is there anything else that I need to be careful with?&lt;/p&gt;</comment>
                            <comment id="2586018" author="jason.chan" created="Thu, 5 Dec 2019 15:49:32 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-43656&quot; title=&quot;Return the timestamp of the top of the oplog in the storage integration layer&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-43656&quot;&gt;&lt;del&gt;SERVER-43656&lt;/del&gt;&lt;/a&gt; implements a method to get the lastAppliedTimestamp through the record store. This requires getting the collection object (and the appropriate collection level locks). This ticket may need to extend the new mechanism added to get the lastAppliedTimestamp while only holding the global lock in MODE_IS. Once this is available, consider replacing &lt;a href=&quot;https://github.com/mongodb/mongo/blob/91fc73a60e523e40defcaa8b6cd808a3091aec34/src/mongo/db/repl/bgsync.cpp#L774&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Helpers::getLast in bgsync.cpp&lt;/a&gt; to use the new interface.&lt;/p&gt;</comment>
                            <comment id="2434935" author="judah.schvimer" created="Thu, 26 Sep 2019 17:39:34 +0000"  >&lt;p&gt;I filed &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-43656&quot; title=&quot;Return the timestamp of the top of the oplog in the storage integration layer&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-43656&quot;&gt;&lt;del&gt;SERVER-43656&lt;/del&gt;&lt;/a&gt; to track the storage integration layer work needed for this.&lt;/p&gt;</comment>
                            <comment id="2434925" author="daniel.gottlieb@10gen.com" created="Thu, 26 Sep 2019 17:33:52 +0000"  >&lt;p&gt;SGTM&lt;/p&gt;</comment>
                            <comment id="2433581" author="judah.schvimer" created="Wed, 25 Sep 2019 21:21:05 +0000"  >&lt;p&gt;Talking to &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.gottlieb&quot; class=&quot;user-hover&quot; rel=&quot;daniel.gottlieb&quot;&gt;daniel.gottlieb&lt;/a&gt;, made me realize that retryable writes are immune to this bug because of session checkout, like transactions. Other &quot;retryable write like&quot; writes that use &lt;tt&gt;setLastOpToSystemLastOpTime&lt;/tt&gt; are still subject to this bug though. &lt;a href=&quot;https://github.com/mongodb/mongo/blob/397f0556a9e2d4fa3106d9490611b405d0e1ecf3/src/mongo/db/commands/set_feature_compatibility_version_command.cpp#L159&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;setFCV&lt;/a&gt; is the simplest example. Sharding also makes use of this in places that, I think, do not always do storage reads.&lt;/p&gt;</comment>
                            <comment id="2433311" author="judah.schvimer" created="Wed, 25 Sep 2019 18:23:26 +0000"  >&lt;p&gt;I&apos;m resuming work on this ticket. &lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;2. WT will error if MongoDB does a read at a timestamp greater than the all_committed.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=agorrod&quot; class=&quot;user-hover&quot; rel=&quot;agorrod&quot;&gt;agorrod&lt;/a&gt;, was this implemented?&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;3.&amp;#93;&lt;/span&gt;2. WT could return the largest timestamp of any update seen in a given WT-transaction. &lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;There are two main uses of &lt;tt&gt;setLastOpToSystemLastOpTime&lt;/tt&gt;.&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Ensuring clients wait for write concern on any writes that could have caused their attempted write to become a &lt;a href=&quot;https://github.com/mongodb/mongo/blob/39daf64f53e984c92d0af615593975a723797f97/src/mongo/db/service_entry_point_mongod.cpp#L140&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&quot;no-op write&quot;&lt;/a&gt;.&lt;/li&gt;
	&lt;li&gt;Ensuring that when clients retry a retryable write (or some other command that is retryable), that they wait for write concern on the original write.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;In the first case there is a WT transaction available, though it has been closed by the time we call &lt;tt&gt;setLastOpToSystemLastOpTime&lt;/tt&gt;. We could expose a way to store the &quot;largest timestamp of any update seen&quot; (maybe on the &lt;tt&gt;OperationContext&lt;/tt&gt;) until it is ready for use. This does not help in the second case though, for retryable writes, where there is no WT transaction, there is just in-memory state indicating there is no work to do. In some cases we have an OpTime to wait on, but in others there&apos;s no clear OpTime to wait on.&lt;/p&gt;

&lt;p&gt;I propose to fix this with the other solution that &quot;The storage integration layer could return the timestamp of the top of the oplog.&quot; We could optimize this in the future for certain cases, but correctness is the problem here, not performance. The other solution just had some potential performance improvements we won&apos;t be able to get. &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.gottlieb&quot; class=&quot;user-hover&quot; rel=&quot;daniel.gottlieb&quot;&gt;daniel.gottlieb&lt;/a&gt;, does this still seem reasonable?&lt;/p&gt;</comment>
                            <comment id="2145976" author="tess.avitabile" created="Tue, 12 Feb 2019 15:42:16 +0000"  >&lt;p&gt;That sounds correct to me. Thank you for writing up the plan, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=judah.schvimer&quot; class=&quot;user-hover&quot; rel=&quot;judah.schvimer&quot;&gt;judah.schvimer&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="2145147" author="judah.schvimer" created="Mon, 11 Feb 2019 21:27:08 +0000"  >&lt;p&gt;As discussed with &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tess.avitabile&quot; class=&quot;user-hover&quot; rel=&quot;tess.avitabile&quot;&gt;tess.avitabile&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=schwerin&quot; class=&quot;user-hover&quot; rel=&quot;schwerin&quot;&gt;schwerin&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=michael.cahill&quot; class=&quot;user-hover&quot; rel=&quot;michael.cahill&quot;&gt;michael.cahill&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=agorrod&quot; class=&quot;user-hover&quot; rel=&quot;agorrod&quot;&gt;agorrod&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=siyuan.zhou&quot; class=&quot;user-hover&quot; rel=&quot;siyuan.zhou&quot;&gt;siyuan.zhou&lt;/a&gt;, and &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.gottlieb&quot; class=&quot;user-hover&quot; rel=&quot;daniel.gottlieb&quot;&gt;daniel.gottlieb&lt;/a&gt;, the ideal fix will be:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;MongoDB will do untimestamped reads for local and majority read concerns (&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-38906&quot; title=&quot;Multi-document transactions should not perform timestamped read ahead of all-committed&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-38906&quot;&gt;&lt;del&gt;SERVER-38906&lt;/del&gt;&lt;/a&gt;).&lt;/li&gt;
	&lt;li&gt;WT will error if MongoDB does a read at a timestamp greater than the &lt;tt&gt;all_committed&lt;/tt&gt;.&lt;/li&gt;
	&lt;li&gt;MongoDB will ask WT for the timestamp it should use instead of &lt;tt&gt;lastApplied&lt;/tt&gt; in &lt;tt&gt;setLastOpToSystemLastOpTime&lt;/tt&gt; (and possibly change the name of the function if appropriate). This could be one of two things:
	&lt;ol&gt;
		&lt;li&gt;The storage integration layer could return the timestamp of the top of the oplog. This should be quite cheap since the timestamp is the &lt;tt&gt;RecordId&lt;/tt&gt; in the oplog and should not require parsing any BSON. This is what is intended by the current broken mechanism.&lt;/li&gt;
		&lt;li&gt;WT could return the largest timestamp of any update seen in a given WT-transaction. This timestamp is being used when waiting for data to be majority committed that&apos;s causally related to a user request. Thus, we only care about the timestamps of updates that the WT-transaction in the request actually saw. This will reduce the amount we have to wait in many cases. We currently ensure that majority reads read data from previously acknowledged majority writes on the same node and connection. To maintain this property, WT will use the &lt;tt&gt;durable_timestamp&lt;/tt&gt; rather than the &lt;tt&gt;commit_timestamp&lt;/tt&gt; of the updates. This prevents the &lt;tt&gt;commitTransaction&lt;/tt&gt; oplog entry from rolling back and an &lt;tt&gt;ignore_prepared=true&lt;/tt&gt; majority read from missing data that was previously read with speculative majority read concern or written with write concern majority (though we are still guaranteed that the transaction will commit at the same &lt;tt&gt;commit_timestamp&lt;/tt&gt; sometime in the future.&lt;/li&gt;
	&lt;/ol&gt;
	&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;If I missed anything or incorrectly summarized the results of the discussion, please correct me.&lt;/p&gt;</comment>
                            <comment id="2137958" author="daniel.gottlieb@10gen.com" created="Tue, 5 Feb 2019 14:05:33 +0000"  >&lt;p&gt;I agree that, on primaries, &quot;last reserved&quot; is effectively the logical clock value. The challenge is on (lagged) secondaries where their logical clock values will know of times arbitrarily far into the future. Because this time is typically used to &quot;wait until&quot;, assigning such a time to stale data would result in unnecessary waiting.&lt;/p&gt;</comment>
                            <comment id="2136856" author="milkie" created="Mon, 4 Feb 2019 17:14:01 +0000"  >&lt;p&gt;The &quot;last reserved optime&quot; seems like an easy route to take; you can just call LogicalClock::getClusterTime().  That works at least on primary nodes.&lt;/p&gt;</comment>
                            <comment id="2136626" author="tess.avitabile" created="Mon, 4 Feb 2019 15:51:32 +0000"  >&lt;p&gt;Discussed this with &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.gottlieb&quot; class=&quot;user-hover&quot; rel=&quot;daniel.gottlieb&quot;&gt;daniel.gottlieb&lt;/a&gt;, and we could consider changing these uses to wait on the clusterTime being majority committed or introduced a new notion of the &quot;last reserved optime&quot;.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="942147">SERVER-43656</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is depended on by">
                                        <issuelink>
            <issuekey id="769446">SERVER-41165</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10520">
                    <name>Problem/Incident</name>
                                            <outwardlinks description="causes">
                                        <issuelink>
            <issuekey id="1117249">SERVER-45800</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="1098672">SERVER-45598</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1024103">SERVER-44820</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="405776">SERVER-30217</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="629826">SERVER-37948</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="682368">SERVER-39356</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="668173">SERVER-38906</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>14.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 4 Feb 2019 17:14:01 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 4 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/SERVER-43656'>SERVER-43656</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 4 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>20.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>daniel.gottlieb@mongodb.com</customfieldvalue>
            <customfieldvalue>milkie@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>jason.chan@mongodb.com</customfieldvalue>
            <customfieldvalue>judah.schvimer@mongodb.com</customfieldvalue>
            <customfieldvalue>lingzhi.deng@mongodb.com</customfieldvalue>
            <customfieldvalue>tess.avitabile@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hul4nb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr671r:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3438">Repl 2019-12-30</customfieldvalue>
    <customfieldvalue id="3439">Repl 2020-01-13</customfieldvalue>
    <customfieldvalue id="3440">Repl 2020-01-27</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hukqwn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>