<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:24:05 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-50949] waitUntilDurable call spanning rollback can set lastDurable ahead of journaling</title>
                <link>https://jira.mongodb.org/browse/SERVER-50949</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;A waitUntilDurable call can span the lifetime of a rollback and cause us to set lastDurable ahead of journaling as follows:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;lastApplied = OpTime(2,2), lastDurable = OpTime(1,1)&lt;/li&gt;
	&lt;li&gt;Asynchronous thread decides to set lastDurable to OpTime(2,2)&lt;/li&gt;
	&lt;li&gt;Rollback occurs. lastApplied = OpTime(1,1), lastDurable = OpTime(1,1)&lt;/li&gt;
	&lt;li&gt;Apply oplog entry. lastApplied = OpTime(1,2), lastDurable = OpTime(1,1)&lt;/li&gt;
	&lt;li&gt;Apply oplog entry. lastApplied = OpTime(3,3), lastDurable = OpTime(1,1)&lt;/li&gt;
	&lt;li&gt;Asynchronous thread succeeds in setting lastDurable to OpTime(2,2). This is not a time in the oplog. Additionally, this implies that OpTime(1,2) is journaled, which it is not.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;I don&apos;t think this can cause us to incorrectly respond to a client that a write is journaled or majority committed. However, it seems possible to violate internal invariants.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1475010">SERVER-50949</key>
            <summary>waitUntilDurable call spanning rollback can set lastDurable ahead of journaling</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="lingzhi.deng@mongodb.com">Lingzhi Deng</assignee>
                                    <reporter username="tess.avitabile@mongodb.com">Tess Avitabile</reporter>
                        <labels>
                    </labels>
                <created>Tue, 15 Sep 2020 14:47:35 +0000</created>
                <updated>Sun, 29 Oct 2023 22:03:10 +0000</updated>
                            <resolved>Thu, 15 Oct 2020 19:19:17 +0000</resolved>
                                                    <fixVersion>4.9.0</fixVersion>
                                    <component>Storage</component>
                                        <votes>0</votes>
                                    <watches>8</watches>
                                                                                                                <comments>
                            <comment id="3447181" author="xgen-internal-githook" created="Thu, 15 Oct 2020 19:16:50 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Lingzhi Deng&apos;, &apos;email&apos;: &apos;lingzhi.deng@mongodb.com&apos;, &apos;username&apos;: &apos;ldennis&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-50949&quot; title=&quot;waitUntilDurable call spanning rollback can set lastDurable ahead of journaling&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-50949&quot;&gt;&lt;del&gt;SERVER-50949&lt;/del&gt;&lt;/a&gt;: Return empty JournalListener token during rollback&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/9f52be8cf511d4fe0b2aa8eb20e18c47a5bf097d&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/9f52be8cf511d4fe0b2aa8eb20e18c47a5bf097d&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3442893" author="tess.avitabile" created="Tue, 13 Oct 2020 18:54:53 +0000"  >&lt;p&gt;That makes sense, and I&apos;m curious to hear what you find!&lt;/p&gt;</comment>
                            <comment id="3437199" author="lingzhi.deng" created="Fri, 9 Oct 2020 16:09:20 +0000"  >&lt;p&gt;Thanks for the info. Now that I looked more into &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-47898&quot; title=&quot;Advancing lastDurable irrespective of lastApplied&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-47898&quot;&gt;&lt;del&gt;SERVER-47898&lt;/del&gt;&lt;/a&gt;, I think I might be wrong. It seems that even if the journal flusher is stopped and restarted, but it is &lt;a href=&quot;https://github.com/mongodb/mongo/blob/1f11a9c73e11ebb6a89b1600a0a03741111c48bc/src/mongo/db/repl/storage_interface_impl.cpp#L1305&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;restarted immediately after &lt;tt&gt;recoverToStableTimestamp&lt;/tt&gt;&lt;/a&gt;. So that means the journal flusher is restarted before the lastApplied is reset. So in the window between then and the reset of lastApplied/lastDurable, there indeed could be an async thread that reads the old lastApplied and spans across the reset of lastApplied/lastDurable. In that case, it could be using an old lastApplied in setting the lastDurable after rollback.&lt;/p&gt;

&lt;p&gt;I think the mistake I had was that I mostly was just reading at the code around recoverToStableTimestamp. While it seems not possible for the journal flusher to span across the entire rollback, but it seems possible for the journal flusher to start after recoverToStableTimestamp and span across the reset of lastApplied/lastDurable. I will verify that.&lt;/p&gt;

&lt;p&gt;If that&apos;s the case, I think we can either reset the lastApplied/lastDurable earlier or restart the journal flusher only after resetting lastApplied/lastDurable.&lt;/p&gt;</comment>
                            <comment id="3437130" author="lingzhi.deng" created="Fri, 9 Oct 2020 15:41:35 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tess.avitabile&quot; class=&quot;user-hover&quot; rel=&quot;tess.avitabile&quot;&gt;tess.avitabile&lt;/a&gt; Can you provide a patch on how to reproduce that? I can probably take a look since I am already in this code. I saw your WIP patch in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-47898&quot; title=&quot;Advancing lastDurable irrespective of lastApplied&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-47898&quot;&gt;&lt;del&gt;SERVER-47898&lt;/del&gt;&lt;/a&gt;. Do you remember which test to run? Thanks.&lt;/p&gt;</comment>
                            <comment id="3436958" author="tess.avitabile" created="Fri, 9 Oct 2020 14:32:27 +0000"  >&lt;p&gt;Yes, this makes sense. &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=dianna.hohensee&quot; class=&quot;user-hover&quot; rel=&quot;dianna.hohensee&quot;&gt;dianna.hohensee&lt;/a&gt; thought perhaps we don&apos;t shut down the JournalFlusher for long enough during rollback, but maybe it shouldn&apos;t matter if the JournalFlusher doesn&apos;t retain any state between stopping and starting, and rollback is holding a global write lock, so no new writes will occur.&lt;/p&gt;

&lt;p&gt;I guess the next person who works on &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-47898&quot; title=&quot;Advancing lastDurable irrespective of lastApplied&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-47898&quot;&gt;&lt;del&gt;SERVER-47898&lt;/del&gt;&lt;/a&gt; will have to look further to explain why advancing lastApplied when we advance lastDurable causes us to violate &lt;a href=&quot;https://github.com/mongodb/mongo/blob/f66f0e54ff6b71000b9a3404d5d9dd43f51f874a/src/mongo/db/repl/topology_coordinator.cpp#L1320-L1322&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this invariant&lt;/a&gt;. What I observed in my patch was that after a rollback, we would attempt to set lastApplied to an OpTime with a higher term and a lower timestamp. This could occur if there&apos;s an asynchronous thread that decides to set lastDurable and spans across rollback, since it could set lastDurable (and lastApplied) to an OpTime ahead of the common point that no longer exists in the oplog after the rollback. Then the next time we set lastApplied as part of secondary oplog application, we would hit the invariant.&lt;/p&gt;</comment>
                            <comment id="3429473" author="lingzhi.deng" created="Tue, 6 Oct 2020 21:21:10 +0000"  >&lt;p&gt;As Dan mentioned about, as the journal flusher &lt;a href=&quot;https://github.com/mongodb/mongo/blob/9ba166c429b07ef3cfb4d8553d39564150844196/src/mongo/db/repl/storage_interface_impl.cpp#L1300&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;gets stopped before rollback&lt;/a&gt; can proceed and only &lt;a href=&quot;https://github.com/mongodb/mongo/blob/9ba166c429b07ef3cfb4d8553d39564150844196/src/mongo/db/repl/storage_interface_impl.cpp#L1305&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;restarted after&lt;/a&gt;, I dont think the scenario is possible with the journal flusher. As part of the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/9cc6fab62d3c66d5ed3517271f6e0fdfa62cd27b/src/mongo/db/storage/control/storage_control.cpp#L91&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;stopStorageControls&lt;/tt&gt;&lt;/a&gt;, we shut down the journal flusher synchronously. Therefore, it is not possible to have an &quot;asynchronous thread decides to set lastDurable to OpTime(2,2)&quot; lingering around across the rollback boundary in the above example.&lt;/p&gt;

&lt;p&gt;That said, there are other callers of &lt;tt&gt;waitUntilDurable()&lt;/tt&gt; that bypass the journal flusher. They are &lt;a href=&quot;https://github.com/mongodb/mongo/blob/9cc6fab62d3c66d5ed3517271f6e0fdfa62cd27b/src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp#L889&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;flushAllFiles()&lt;/tt&gt;&lt;/a&gt; and &lt;a href=&quot;https://github.com/mongodb/mongo/blob/9cc6fab62d3c66d5ed3517271f6e0fdfa62cd27b/src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.cpp#L280&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;waitUntilUnjournaledWritesDurable()&lt;/tt&gt;&lt;/a&gt;. And technically these two could be spanning rollbacks.&lt;/p&gt;

&lt;p&gt;For &lt;tt&gt;flushAllFiles()&lt;/tt&gt;, its currently called in fsync (&lt;a href=&quot;https://github.com/mongodb/mongo/blob/9cc6fab62d3c66d5ed3517271f6e0fdfa62cd27b/src/mongo/db/commands/fsync.cpp#L150&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;errmsgRun&lt;/tt&gt;&lt;/a&gt; and &lt;a href=&quot;https://github.com/mongodb/mongo/blob/9cc6fab62d3c66d5ed3517271f6e0fdfa62cd27b/src/mongo/db/commands/fsync.cpp#L375&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;FSyncLockThread::run&lt;/tt&gt;&lt;/a&gt;), &lt;a href=&quot;https://github.com/mongodb/mongo/blob/9cc6fab62d3c66d5ed3517271f6e0fdfa62cd27b/src/mongo/db/repair.cpp#L85&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;rebuildIndexesForNamespace&lt;/tt&gt;&lt;/a&gt;, &lt;a href=&quot;https://github.com/mongodb/mongo/blob/9cc6fab62d3c66d5ed3517271f6e0fdfa62cd27b/src/mongo/db/repl/oplog.cpp#L632&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;createOplog()&lt;/tt&gt;&lt;/a&gt; and &lt;a href=&quot;https://github.com/mongodb/mongo/blob/9cc6fab62d3c66d5ed3517271f6e0fdfa62cd27b/src/mongo/db/write_concern.cpp#L310&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;fsync option for writeConcern&lt;/a&gt;. The fsync command, repair and oplog creation all take a global lock at last in IS mode. And I believe the &quot;fsync option for writeConcern&quot; is no longer used? Since the rollback takes global write lock, I don&apos;t think any of them can span across rollback.&lt;/p&gt;

&lt;p&gt;For &lt;tt&gt;waitUntilUnjournaledWritesDurable()&lt;/tt&gt;, it is currently called in &lt;a href=&quot;https://github.com/mongodb/mongo/blob/9cc6fab62d3c66d5ed3517271f6e0fdfa62cd27b/src/mongo/db/repl/replication_coordinator_external_state_impl.cpp#L465&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;initializeReplSetStorage&lt;/tt&gt;&lt;/a&gt;, &lt;a href=&quot;https://github.com/mongodb/mongo/blob/9cc6fab62d3c66d5ed3517271f6e0fdfa62cd27b/src/mongo/db/repl/replication_recovery.cpp#L543&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;ReplicationRecoveryImpl::_recoverFromUnstableCheckpoint&lt;/tt&gt;&lt;/a&gt;(as part of &lt;tt&gt;recoverFromOplog&lt;/tt&gt;) and in &lt;tt&gt;rollback_internal::syncFixUp&lt;/tt&gt; (during RollbackViaRefetch). We dont need to worry about &lt;tt&gt;initializeReplSetStorage&lt;/tt&gt;. And &lt;tt&gt;recoverFromOplog&lt;/tt&gt; is called either from start up repl recovery or from RollbackImpl::runRollback. For both &lt;tt&gt;recoverFromOplog&lt;/tt&gt; in RTT and &lt;tt&gt;rollback_internal::syncFixUp&lt;/tt&gt; in RollbackViaRefetch, we call &lt;tt&gt;resetLastOpTimesFromOplog&lt;/tt&gt; at the end of rollback and thus we don&apos;t need to worry about the &lt;tt&gt;waitUntilUnjournaledWritesDurable()&lt;/tt&gt; calls during rollback because the lastDurable and lastApplied will be reset before the rollback finishes.&lt;/p&gt;

&lt;p&gt;Therefore, based on the code auditing above, I don&apos;t think it is possible for an &quot;asynchronous thread that decides to set lastDurable&quot; to span across rollback.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tess.avitabile&quot; class=&quot;user-hover&quot; rel=&quot;tess.avitabile&quot;&gt;tess.avitabile&lt;/a&gt; &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=dianna.hohensee&quot; class=&quot;user-hover&quot; rel=&quot;dianna.hohensee&quot;&gt;dianna.hohensee&lt;/a&gt; Does the above make sense? But I could be missing something too.&lt;/p&gt;</comment>
                            <comment id="3397004" author="dianna.hohensee" created="Wed, 16 Sep 2020 15:16:09 +0000"  >&lt;p&gt;The JournalFlusher doesn&apos;t get stopped on stepdown, we just play a dance to reset its behavior to non-primary mode, via a &lt;a href=&quot;https://github.com/mongodb/mongo/blob/547823d400ce7ada24e3dde37f07e59845260ffa/src/mongo/db/repl/replication_coordinator_external_state_impl.cpp#L792-L828&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;relatively complicated interruption sequence on stepdown&lt;/a&gt;. The JournalFlusher is only &lt;a href=&quot;https://github.com/mongodb/mongo/blob/547823d400ce7ada24e3dde37f07e59845260ffa/src/mongo/db/repl/storage_interface_impl.cpp#L1299-L1307&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;shutdown and restarted momentarily for StorageEngine::recoverToStableTimestamp&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;I&apos;ve moved the JournalFlusher to be conceptually above the storage layer, to control the storage layer but not be a part of it. The argument is that the JournalFlusher exists to help replication manage the storage engine for performance gains, and ensure users don&apos;t hurt themselves by never doing j:true writes &#8211; I could word better with more thought&#160;&lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&#160;Therefore, we still need the RecoveryUnit::waitUntilDurable() access point. Though it might be nice to discourage its use somehow.&lt;/p&gt;</comment>
                            <comment id="3396941" author="daniel.gottlieb@10gen.com" created="Wed, 16 Sep 2020 15:05:57 +0000"  >&lt;blockquote&gt;
&lt;p&gt;I&apos;ve moved all the waitUntilDurable() callers onto the JournalFlusher recently.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Ah interesting. Should we remove &lt;tt&gt;waitUntilDurable&lt;/tt&gt; from the recovery unit to make sure no new callers get introduced?&lt;/p&gt;

&lt;p&gt;Second:&lt;br/&gt;
If all the callers are in the journal flusher, and the journal flusher gets stopped before rollback can proceed and only restarted after, how is this scenario possible? Are we restarting the journal flusher before the oplog truncation part of rollback is performed (&lt;tt&gt;cappedTruncateAfter&lt;/tt&gt; I believe)?&lt;/p&gt;</comment>
                            <comment id="3396922" author="dianna.hohensee" created="Wed, 16 Sep 2020 14:58:46 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.gottlieb&quot; class=&quot;user-hover&quot; rel=&quot;daniel.gottlieb&quot;&gt;daniel.gottlieb&lt;/a&gt;&#160;I&apos;ve moved all the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/728fc1c52300cd409dd0619ebe669f8d0bd1835a/src/mongo/db/storage/control/journal_flusher.h#L89-L100&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;waitUntilDurable() callers onto the JournalFlusher&lt;/a&gt; recently.&#160;flushAllFiles() and&#160;waitUntilUnjournaledWritesDurable() calls are the only ways to bypass the JournalFlusher now, the latter of which is only called during repl rollback/recovery. We moved callers onto the JournalFlusher, so we could ensure interruption of setting the oplogTruncateAfterPoint on stepdown, because we want to clear the&#160;oplogTruncateAfterPoint on stepdown.&lt;/p&gt;</comment>
                            <comment id="3396714" author="daniel.gottlieb@10gen.com" created="Wed, 16 Sep 2020 14:25:04 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=dianna.hohensee&quot; class=&quot;user-hover&quot; rel=&quot;dianna.hohensee&quot;&gt;dianna.hohensee&lt;/a&gt; can you clarify what part of &lt;a href=&quot;https://github.com/mongodb/mongo/blob/5a1953c4ae4416a60c45c5c90a210bd84b352032/src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp#L344-L355&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;waitUntilDurable&lt;/tt&gt;&lt;/a&gt; synchronizes with whether the &lt;tt&gt;JournalFlusher&lt;/tt&gt; is running? If the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/5a1953c4ae4416a60c45c5c90a210bd84b352032/src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp#L354&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;onDurable&lt;/tt&gt;&lt;/a&gt; call is what&apos;s advancing &lt;tt&gt;lastDurable&lt;/tt&gt;, it seems that turning the journal flusher off would still allow for the &lt;tt&gt;lastDurable&lt;/tt&gt; time to be advanced.&lt;/p&gt;</comment>
                            <comment id="3395277" author="dianna.hohensee" created="Tue, 15 Sep 2020 19:06:13 +0000"  >&lt;p&gt;We should investigate when during rollback lastApplied is unset/reset, and what opportunities the JournalFlusher has to pick up a lastApplied timestamp value before the reset. The JournalFlusher is turned off and on around calling recoverToAStableTimestamp on the storage engine. A solution might be to turn the JournalFlusher off for a larger duration of rollback, so it can&apos;t asynchronously grab anything mischievous.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                        <issuelink>
            <issuekey id="1336424">SERVER-47898</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>11.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 15 Sep 2020 19:06:13 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 16 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 16 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>daniel.gottlieb@mongodb.com</customfieldvalue>
            <customfieldvalue>dianna.hohensee@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>lingzhi.deng@mongodb.com</customfieldvalue>
            <customfieldvalue>tess.avitabile@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hy5z1j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hxsfyf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="4215">Execution Team 2020-10-19</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hy5lav:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>