<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:15:33 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-47898] Advancing lastDurable irrespective of lastApplied</title>
                <link>https://jira.mongodb.org/browse/SERVER-47898</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;lastDurable is set asynchronously from lastApplied. This can cause us to attempt to advance lastDurable beyond lastApplied. When we attempt to advance lastDurable beyond lastApplied, &lt;a href=&quot;https://github.com/mongodb/mongo/blob/3390cf27165d49ad7739b447b50927e874ec2c1e/src/mongo/db/repl/member_data.cpp#L162&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;we skip advancing lastDurable&lt;/a&gt;. This can increase latency for w:&quot;majority&quot; writes, which wait for lastDurable to advance on a majority. It also causes odd behavior, where w:1,j:true writes return success, but &lt;tt&gt;durableOpTime&lt;/tt&gt; in &lt;tt&gt;replSetGetStatus&lt;/tt&gt; is not updated. When we attempt to advance lastDurable beyond lastApplied, we should advance lastDurable .&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Original description:&lt;/p&gt;

&lt;p&gt;The comments surrounding this log line suggest that this is unexpected behavior:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/r4.4.0-rc3/src/mongo/db/repl/member_data.cpp#L163-L173&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/r4.4.0-rc3/src/mongo/db/repl/member_data.cpp#L163-L173&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Observed on an internal cluster with a primary running 4.4.0-rc3 and secondaries on 4.2.1. Happy to increase logging level if it would be helpful to understand the behavior here.&lt;/p&gt;

&lt;p&gt;cc &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tess.avitabile&quot; class=&quot;user-hover&quot; rel=&quot;tess.avitabile&quot;&gt;tess.avitabile&lt;/a&gt;&lt;/p&gt;</description>
                <environment></environment>
        <key id="1336424">SERVER-47898</key>
            <summary>Advancing lastDurable irrespective of lastApplied</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="kishore.devireddy@mongodb.com">Kishore Devireddy</assignee>
                                    <reporter username="kelsey.schubert@mongodb.com">Kelsey Schubert</reporter>
                        <labels>
                            <label>PM-3489-Milestone-MiscImprovement-CP</label>
                            <label>former-quick-wins</label>
                    </labels>
                <created>Fri, 1 May 2020 20:39:05 +0000</created>
                <updated>Thu, 25 Jan 2024 20:44:39 +0000</updated>
                            <resolved>Mon, 11 Dec 2023 22:29:32 +0000</resolved>
                                                    <fixVersion>7.3.0-rc0</fixVersion>
                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>20</watches>
                                                                                                                <comments>
                            <comment id="5942349" author="xgen-internal-githook" created="Mon, 11 Dec 2023 22:32:13 +0000"  >&lt;p&gt;Author: &lt;/p&gt;
{&apos;name&apos;: &apos;Kishore Devireddy&apos;, &apos;email&apos;: &apos;kishore.devireddy@mongodb.com&apos;, &apos;username&apos;: &apos;kishorekrd&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-47898&quot; title=&quot;Advancing lastDurable irrespective of lastApplied&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-47898&quot;&gt;&lt;del&gt;SERVER-47898&lt;/del&gt;&lt;/a&gt; Advancing lastDurable irrespective of lastApplied&lt;/p&gt;

&lt;p&gt;GitOrigin-RevId: 9c3067d69a4df40ef9cad9e284000e5b236c216e&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/11e4b5c0f29a45fd2928ca75b8c386900aef440b&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/11e4b5c0f29a45fd2928ca75b8c386900aef440b&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="5942345" author="JIRAUSER1275243" created="Mon, 11 Dec 2023 22:29:33 +0000"  >&lt;p&gt;&lt;a href=&quot;https://spruce.mongodb.com/version/6577600cd1fe07b644d7210c/tasks?sorts=STATUS%3AASC%3BBASE_STATUS%3ADESC&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://spruce.mongodb.com/version/6577600cd1fe07b644d7210c/tasks?sorts=STATUS%3AASC%3BBASE_STATUS%3ADESC&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3456239" author="xgen-internal-githook" created="Wed, 21 Oct 2020 14:08:44 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Max Hirschhorn&apos;, &apos;email&apos;: &apos;max.hirschhorn@mongodb.com&apos;, &apos;username&apos;: &apos;visemet&apos;}
&lt;p&gt;Message: Revert &quot;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-47898&quot; title=&quot;Advancing lastDurable irrespective of lastApplied&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-47898&quot;&gt;&lt;del&gt;SERVER-47898&lt;/del&gt;&lt;/a&gt;: Advancing lastDurable should advance lastApplied&quot;&lt;/p&gt;

&lt;p&gt;This reverts commit 2ebea0648a4bd2d31abb5251d66b2b7925bb06a8.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/c997b4d7ea671a08afba602126a65bea53649e7d&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/c997b4d7ea671a08afba602126a65bea53649e7d&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3447183" author="xgen-internal-githook" created="Thu, 15 Oct 2020 19:16:53 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Lingzhi Deng&apos;, &apos;email&apos;: &apos;lingzhi.deng@mongodb.com&apos;, &apos;username&apos;: &apos;ldennis&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-47898&quot; title=&quot;Advancing lastDurable irrespective of lastApplied&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-47898&quot;&gt;&lt;del&gt;SERVER-47898&lt;/del&gt;&lt;/a&gt;: Advancing lastDurable should advance lastApplied&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/2ebea0648a4bd2d31abb5251d66b2b7925bb06a8&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/2ebea0648a4bd2d31abb5251d66b2b7925bb06a8&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3397710" author="tess.avitabile" created="Wed, 16 Sep 2020 19:26:58 +0000"  >&lt;blockquote&gt;
&lt;p&gt;Are we saying that lastDurable won&apos;t be advanced during oplog application until after the batch completes?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Yes, today we advance both lastDurable and lastApplied after the batch is applied. In the future, we could consider separating setting lastDurable and lastApplied in order to acknowledge writes faster, but we don&apos;t do this today.&lt;/p&gt;</comment>
                            <comment id="3396958" author="dianna.hohensee" created="Wed, 16 Sep 2020 15:10:43 +0000"  >&lt;p&gt;The&#160;oplogTruncateAfterPoint updates are independent. We update the&#160;oplogTruncateAfterPoint with WT&apos;s all_durable timestamp prior to the journal flush. Then we try to update lastDurable after the journal flush, with the added criteria that lastDurable can only be updated with a real oplog timestamp, so we read an oplog entry &amp;lt;= all_durable with which to attempt to set repl&apos;s lastDurable.&lt;/p&gt;

&lt;p&gt;The&#160;oplogTruncateAfterPoint does not have the criteria of a real oplog timestamp, because WT&apos;s all_durable doesn&apos;t, and oplog truncation works with that &#8211; truncation used to be inclusive, so the +1 recovery trick, I assumed motivated this.&lt;/p&gt;</comment>
                            <comment id="3396956" author="daniel.gottlieb@10gen.com" created="Wed, 16 Sep 2020 15:10:33 +0000"  >&lt;blockquote&gt;
&lt;p&gt; it would be more straight-forward to allow bumping lastDurable to bump lastApplied, in order to address the concern with majority write performance.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;How should I think about what &lt;tt&gt;lastDurable&lt;/tt&gt; and &lt;tt&gt;lastApplied&lt;/tt&gt; mean on secondaries? Particularly when the system is in the state where oplog entries have been written, but the data writes are incomplete. Are we saying that &lt;tt&gt;lastDurable&lt;/tt&gt; won&apos;t be advanced during oplog application until after the batch completes? (maybe that&apos;s already true somehow). Are we going to bump &lt;tt&gt;lastApplied&lt;/tt&gt; even though the data writes aren&apos;t done? (and presumably re-work how secondary reads at lastApplied need to work).&lt;/p&gt;</comment>
                            <comment id="3396907" author="tess.avitabile" created="Wed, 16 Sep 2020 14:55:20 +0000"  >&lt;p&gt;I think it would make sense to let lastDurable and lastApplied be independent and relax our rule that lastDurable &amp;lt;= lastApplied, particularly for the consensus arbiters use case. However, I&apos;m not certain where we make this assumption, so relaxing it would be a larger investigation. Since in our current system, it&apos;s safe to assume lastDurable &amp;lt;= lastApplied, it would be more straight-forward to allow bumping lastDurable to bump lastApplied, in order to address the concern with majority write performance.&lt;/p&gt;

&lt;p&gt;I&apos;m not certain whether durableOpTime is one-to-one with oplogTruncateAfterPoint.&lt;/p&gt;</comment>
                            <comment id="3396587" author="daniel.gottlieb@10gen.com" created="Wed, 16 Sep 2020 13:46:28 +0000"  >&lt;p&gt;Apologies if this is the wrong place to ask:&lt;/p&gt;

&lt;p&gt;When I think of &lt;tt&gt;lastDurable&lt;/tt&gt; value, the guarantee in my mental model is that all oplog timestamps at that time and earlier are accounted for and persisted on disk with an &lt;tt&gt;fsync&lt;/tt&gt; in write-ahead-log files.&lt;/p&gt;

&lt;p&gt;When I think of &lt;tt&gt;lastApplied&lt;/tt&gt;, I see that as some relaxed notion (due to holes on primaries) of what operations are reflected in the data that a (standard, non-oplog) reader can see.&lt;/p&gt;

&lt;p&gt;In our implementation, on primaries, the relationship makes sense that &lt;tt&gt;lastDurable&lt;/tt&gt; advancing can also advance &lt;tt&gt;lastApplied&lt;/tt&gt; as the data write is atomic with the oplog write. But on secondaries, the relationship (based on my definition) is not necessarily so. The oplog writes happen first and &lt;tt&gt;lastDurable&lt;/tt&gt; can be technically be advanced (in an effort to minimize the latency for moving the majority point) without applying the data writes. I&apos;m not sure if our code does that.&lt;/p&gt;

&lt;p&gt;When I think of the arbiters with oplog project (maybe no longer a real project we plan on doing?), it&apos;s natural for those theoretical processes to have a &lt;tt&gt;lastDurable&lt;/tt&gt; (they can vote on majority points) but not a &lt;tt&gt;lastApplied&lt;/tt&gt; (they cannot service data reads, just oplog reads).&lt;/p&gt;

&lt;p&gt;Sorry for the words, here&apos;s an actual question: Why is the right thing to do also advance &lt;tt&gt;lastApplied&lt;/tt&gt; in-toe with &lt;tt&gt;lastDurable&lt;/tt&gt; as opposed to letting &lt;tt&gt;lastDurable&lt;/tt&gt; be independent (i.e: relax our rules/invariants regarding lastDurable &amp;gt; lastApplied)? If independence isn&apos;t suitable, can someone provide a better way for me to think about what &lt;tt&gt;lastApplied&lt;/tt&gt; and &lt;tt&gt;lastDurable&lt;/tt&gt; represent?&lt;/p&gt;

&lt;p&gt;Third question related to:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;When we attempt to advance lastDurable beyond lastApplied, we skip advancing lastDurable. &amp;lt;snip&amp;gt; It also causes odd behavior, where w:1,j:true writes return success, but durableOpTime in replSetGetStatus is not updated.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Is the &lt;tt&gt;durableOpTime&lt;/tt&gt; 1 to 1 with &lt;tt&gt;oplogTruncateAfterPoint&lt;/tt&gt;? If we skip advancing (&quot;hold back&quot;) the &lt;tt&gt;durableOpTime&lt;/tt&gt; will we also hold back the &lt;tt&gt;oplogTruncateAfterPoint&lt;/tt&gt;?&lt;/p&gt;</comment>
                            <comment id="3395350" author="tess.avitabile" created="Tue, 15 Sep 2020 19:41:00 +0000"  >&lt;p&gt;Thanks for thinking this through with me!&lt;/p&gt;</comment>
                            <comment id="3395266" author="dianna.hohensee" created="Tue, 15 Sep 2020 19:01:52 +0000"  >&lt;p&gt;Ah, I see finally. Thanks, that makes sense.&lt;/p&gt;</comment>
                            <comment id="3395159" author="tess.avitabile" created="Tue, 15 Sep 2020 18:18:34 +0000"  >&lt;p&gt;The bug I described is harder to hit without making the optimization, but I believe it still exists today. In my example, I didn&apos;t assume that advancing lastDurable would advance lastApplied.&lt;/p&gt;</comment>
                            <comment id="3394716" author="dianna.hohensee" created="Tue, 15 Sep 2020 15:35:54 +0000"  >&lt;p&gt;It looks like today we won&apos;t forward the lastDurable timestamp if the new value is ahead of lastApplied: that check is done &lt;a href=&quot;https://github.com/mongodb/mongo/blob/f11e900c83a67aca01b1d681e35c05a5031fe810/src/mongo/db/repl/member_data.cpp#L155-L160&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here in MemberData&lt;/a&gt;. So I think there isn&apos;t a bug right now, but there would be if we tried to move lastApplied forward with lastDurable as an optimization?&lt;/p&gt;

&lt;p&gt;I think the original premise of this ticket was because of that odd log message in PRIMARY state that we didn&apos;t understand at the time. Moving lastApplied forward with lastDurable in state PRIMARY would be a new optimization because of the Replicate Before Journaling changes, but such an optimization is blocked by incorrect things happening in ROLLBACK if it were done for all states right now.&lt;/p&gt;</comment>
                            <comment id="3394542" author="tess.avitabile" created="Tue, 15 Sep 2020 14:44:18 +0000"  >&lt;p&gt;Thanks for that analysis! If it is the case that a waitUntilDurable call can span the lifetime of a rollback, then even without the optimization in this ticket, I think we can get into the scenario I outlined above, which might violate internal invariants:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;lastApplied = OpTime(2,2), lastDurable = OpTime(1,1)&lt;/li&gt;
	&lt;li&gt;Asynchronous thread decides to set lastDurable to OpTime(2,2)&lt;/li&gt;
	&lt;li&gt;Rollback occurs. lastApplied = OpTime(1,1), lastDurable = OpTime(1,1)&lt;/li&gt;
	&lt;li&gt;Apply oplog entry. lastApplied = OpTime(1,2), lastDurable = OpTime(1,1)&lt;/li&gt;
	&lt;li&gt;Apply oplog entry. lastApplied = OpTime(3,3), lastDurable = OpTime(1,1)&lt;/li&gt;
	&lt;li&gt;Asynchronous thread succeeds in setting lastDurable to OpTime(2,2). This is not a time in the oplog. Additionally, this implies that OpTime(1,2) is journaled, which it is not.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;I think this scenario would be a bug (and from what you said, it sounds like a beginning-of-time bug for the JournalFlusher). I think I&apos;ll file this as a bug on Execution and mark this ticket as dependent on that one.&lt;/p&gt;</comment>
                            <comment id="3394213" author="dianna.hohensee" created="Tue, 15 Sep 2020 12:52:35 +0000"  >&lt;p&gt;So the only scenario I can think of to cause the problem is the JournalFlusher thread simply running asynchronously in non-PRIMARY state: starting waitUntilDurable() and &lt;a href=&quot;https://github.com/mongodb/mongo/blob/4d5c5fefbd81ee63831b4264de872e2338875fdd/src/mongo/db/repl/replication_coordinator_external_state_impl.cpp#L1075&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;fetching a timestamp via the JournalListener&lt;/a&gt;(lastApplied), and then not finishing and not &lt;a href=&quot;https://github.com/mongodb/mongo/blob/4d5c5fefbd81ee63831b4264de872e2338875fdd/src/mongo/db/repl/replication_coordinator_external_state_impl.cpp#L1079&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;calling JournalListener::onDurable&lt;/a&gt; to update lastDurable until after rollback finishes / clears the lastApplied timestamp.&lt;/p&gt;

&lt;p&gt;I think this is the scenario you proposed back on August 14th. I think I was lost in the weeds of the other bugs and I didn&apos;t analyze and answer properly. I don&apos;t think what you laid out is a bug, though. It seems like it should always have been the case that this can happen, as long as there&apos;s been a JournalFlusher and rollback. The JournalFlusher takes lastApplied, flushes and tries to set lastDurable with it. I wouldn&apos;t be surprised if the logic to skip updating lastDurable if ahead of lastApplied was deliberately done to solve the rollback problem you found when not skipping.&lt;/p&gt;

&lt;p&gt;Looks like&#160;you &lt;a href=&quot;https://github.com/mongodb/mongo/blob/e27815102992d997a8db431a6916da982b5d9315/src/mongo/db/repl/member_data.cpp#L155-L157&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;documented this aspect a bit ago&lt;/a&gt; in the related ticket, though not because of rollback per se: JournalListener::onDurable led me down to MemberData::setLastDurableOpTimeAndWallTime.&lt;/p&gt;

&lt;p&gt;Rollback seems like the only scenario where forwarding the lastApplied along with lastDurable is unsafe. PRIMARY state should be be safe, because we aren&apos;t manually manipulating the lastApplied value anywhere &#8211; that I know of. Secondary batch application should not run into any problems because in that repl state the JournalFlusher only grabs lastApplied to update lastDurable with.&lt;/p&gt;

&lt;p&gt;&#160;The log Kelsey identifies in the first comment&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;ctx&quot;:&quot;WTJournalFlusher&quot;,&quot;msg&quot;:&quot;Durable progress is ahead of the applied progress. This is likely due to a rollback&quot;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;seems to have shown up because PRIMARY state started getting ahead of lastApplied like only rollback used to do.&lt;/p&gt;

&lt;p&gt;You could theoretically try turninhg the JournalFlusher off for more of rollback: &lt;a href=&quot;https://github.com/mongodb/mongo/blob/e27815102992d997a8db431a6916da982b5d9315/src/mongo/db/repl/storage_interface_impl.cpp#L1302-L1307&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;we turn it off briefly around storage engine recovery to a stable timestamp&lt;/a&gt;. SECONDARY state wouldn&apos;t ever try to move lastDurable ahead of lastApplied; rollback wouldn&apos;t try if the JournalFlusher were off, except for during the explicit waitUntilUnjournaledWritesDurable() calls I see in rs_rollback.cpp and replication_recovery.cpp. I&apos;m not familiar with whether we try to forward the lastDurable timestamp in any other code paths than waitUntilDurable().&lt;/p&gt;</comment>
                            <comment id="3387113" author="tess.avitabile" created="Fri, 11 Sep 2020 13:32:00 +0000"  >&lt;p&gt;That&apos;s correct, that lastDurableBeingSet would have a greater term than lastApplied but lesser timestamp. This could happen if the node that rolled back was at a later timestamp than the new primary. For example:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;lastApplied=Optime(1,3), lastDurable=Optime(1,2)&lt;/li&gt;
	&lt;li&gt;Rollback lastApplied and lastDurable to common point OpTime(1,1)&lt;/li&gt;
	&lt;li&gt;Asynchronous thread sets lastDurable and lastApplied to OpTime(1,3)&lt;/li&gt;
	&lt;li&gt;Oplog application sets lastApplied to OpTime(2,2)&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;This bug could occur if we allow asynchronous threads to set lastDurable while the node is in state SECONDARY. Do you think it&apos;s still possible for this to occur?&lt;/p&gt;</comment>
                            <comment id="3385795" author="dianna.hohensee" created="Thu, 10 Sep 2020 17:25:02 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tess.avitabile&quot; class=&quot;user-hover&quot; rel=&quot;tess.avitabile&quot;&gt;tess.avitabile&lt;/a&gt;, could you explain what that &lt;a href=&quot;https://github.com/mongodb/mongo/blob/f66f0e54ff6b71000b9a3404d5d9dd43f51f874a/src/mongo/db/repl/topology_coordinator.cpp#L1320-L1322&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;invariant&lt;/a&gt;&#160;means?&lt;/p&gt;

&lt;p&gt;It seems like to get into the if-statement, the lastDurableBeingSet, &lt;tt&gt;opTimeAndWallTime.opTime&lt;/tt&gt;, is != lastApplied.&lt;br/&gt;
 Then getting passed the first invariant means that lastDurableBeingSet is newer than lastApplied.&lt;br/&gt;
 Then I&apos;m guessing lastApplied and the lastDurableBeingSet have initialized terms, so the invariant fails because lastDurableBeingSet is suddenly not newer than lastApplied?&lt;/p&gt;

&lt;p&gt;I don&apos;t get why&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;invariant(opTime &amp;gt; myLastAppliedOpTime)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;passes, but&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;invariant(opTime.getTimestamp() &amp;gt; myLastAppliedOpTime.getTimestamp())&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;would ever not pass after that?&lt;/p&gt;

&lt;p&gt;&lt;b&gt;UPDATE&lt;/b&gt;&lt;br/&gt;
Oh, it must be the term in the Optime comparison that&apos;s tripping. So lastDurableBeingSet has a greater term value than lastApplied, but lesser timestamp??&lt;/p&gt;</comment>
                            <comment id="3383734" author="tess.avitabile" created="Wed, 9 Sep 2020 18:31:23 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=dianna.hohensee&quot; class=&quot;user-hover&quot; rel=&quot;dianna.hohensee&quot;&gt;dianna.hohensee&lt;/a&gt;, I rebased and tested by patch again, but I encountered the same&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/f66f0e54ff6b71000b9a3404d5d9dd43f51f874a/src/mongo/db/repl/topology_coordinator.cpp#L1320-L1322&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;invariant&lt;/a&gt; after rollback. Do you have any guess as to why this might happen? My understanding is that this is unexpected, since after&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-48149&quot; title=&quot;Move callers of waitUntilDurable onto JournalFlusher::waitForJournalFlush&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-48149&quot;&gt;&lt;del&gt;SERVER-48149&lt;/del&gt;&lt;/a&gt;, there shouldn&apos;t be updates to lastDurable that are asynchronous with replica set state transition.&lt;/p&gt;</comment>
                            <comment id="3367321" author="tess.avitabile" created="Mon, 31 Aug 2020 13:45:44 +0000"  >&lt;p&gt;Thanks, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=dianna.hohensee&quot; class=&quot;user-hover&quot; rel=&quot;dianna.hohensee&quot;&gt;dianna.hohensee&lt;/a&gt;! I&apos;ll reschedule this ticket.&lt;/p&gt;</comment>
                            <comment id="3364798" author="dianna.hohensee" created="Fri, 28 Aug 2020 19:11:31 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-48149&quot; title=&quot;Move callers of waitUntilDurable onto JournalFlusher::waitForJournalFlush&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-48149&quot;&gt;&lt;del&gt;SERVER-48149&lt;/del&gt;&lt;/a&gt; has been fixed, so I think this task should now be unblocked.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-49695&quot; title=&quot;Clarify and correct synchronization of isOplogTruncateAfterPointBeingUsedForPrimary&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-49695&quot;&gt;&lt;del&gt;SERVER-49695&lt;/del&gt;&lt;/a&gt;, after investigating, turned out to be a problem with &lt;a href=&quot;https://github.com/mongodb/mongo/blob/7bc9d6f843b1ca2df2bdc00a895d825acda4f446/src/mongo/db/repl/replication_consistency_markers_impl.cpp#L374-L392&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;fassert&apos;ing in the &lt;tt&gt;ReplicationConsistencyMarkers::setOplogTruncateAfterPoint&lt;/tt&gt;&lt;/a&gt; codepath, when we should be handling the error. I don&apos;t think it should pose a problem for you, the BFGs are rare, and the fassert cares about timing not timestamps.&lt;/p&gt;</comment>
                            <comment id="3356108" author="dianna.hohensee" created="Mon, 24 Aug 2020 19:13:30 +0000"  >&lt;p&gt;Yes, I think that&apos;s possible, and is the bug scenario. A direct waitUntilDurable call that doesn&apos;t go through the JournalFlusher thread can get far enough to fetch a truncatePoint timestamp X (getToken), then stepdown runs, and then try to set the durableTimestamp with that prefetched timestamp X.&lt;/p&gt;</comment>
                            <comment id="3070025" author="tess.avitabile" created="Tue, 5 May 2020 18:22:19 +0000"  >&lt;p&gt;Skipping update of lastDurable shouldn&apos;t mess up write concern confirmations, or make unusual behavior in what can be read. If I understand the code correctly, j:true writes will call &lt;tt&gt;setLastDurableOpTimeAndWallTime()&lt;/tt&gt;, but they don&apos;t wait for lastDurable to advance. This will be a little odd, since &lt;tt&gt;durableOpTime&lt;/tt&gt; in the &lt;tt&gt;replSetGetStatus&lt;/tt&gt; response won&apos;t be updated promptly when a j:true write succeeds. But this means there is no performance impact for j:true writes. Since we don&apos;t have a readConcern level like &quot;only show journaled data&quot;, there&apos;s no impact on what can be read.&lt;/p&gt;

&lt;p&gt;There would be a performance impact for w:&quot;majority&quot; writes, which do wait for the lastDurable to advance on a majority.&lt;/p&gt;

&lt;p&gt;I think I agree that we should do this on master only.&lt;/p&gt;</comment>
                            <comment id="3069674" author="dianna.hohensee" created="Tue, 5 May 2020 16:09:15 +0000"  >&lt;p&gt;Is skipping update of the lastDurable timestamp going to mess up any write concern confirmations, or make unusual behavior in what can be read? waitUntilDurable, which updates lastDurable via the JournalListener, is typically called for j:true writes and otherwise on a periodic cadence by the JournalFlusher.&lt;/p&gt;

&lt;p&gt;In terms of correctness of oplog truncation via the oplogTruncateAfterPoint, that logic is safe because the&#160;oplogTruncateAfterPoint gets updated regardless of what happens with the lastDurable timestamp update attempt. And oplog visibility via oplogReadTimestamp is a separate system.&lt;/p&gt;</comment>
                            <comment id="3069049" author="judah.schvimer" created="Tue, 5 May 2020 12:52:16 +0000"  >&lt;p&gt;I think leaving 4.4 as is and making the change on 4.5+, which will have a lot of time to burn in, would be a worthwhile perf gain.&lt;/p&gt;</comment>
                            <comment id="3069041" author="tess.avitabile" created="Tue, 5 May 2020 12:43:14 +0000"  >&lt;p&gt;I noticed that &lt;a href=&quot;https://github.com/mongodb/mongo/blob/r4.4.0-rc3/src/mongo/db/repl/member_data.cpp#L160&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;we don&apos;t set lastDurable forward if it would move ahead of lastApplied&lt;/a&gt;, so we maintain that lastDurable is never ahead of lastApplied. This means there&apos;s no concern about sending heartbeats that have lastDurable ahead of lastApplied. We could consider changing the code to advance both lastApplied and lastDurable when we try to advance lastDurable ahead of lastApplied, or we could leave the code as is. It&apos;s possible there would be a performance benefit to w:&quot;majority&quot; writes if we make the change. My inclination is to leave the code as is, at least on the 4.4 branch.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=dianna.hohensee&quot; class=&quot;user-hover&quot; rel=&quot;dianna.hohensee&quot;&gt;dianna.hohensee&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=judah.schvimer&quot; class=&quot;user-hover&quot; rel=&quot;judah.schvimer&quot;&gt;judah.schvimer&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=siyuan.zhou&quot; class=&quot;user-hover&quot; rel=&quot;siyuan.zhou&quot;&gt;siyuan.zhou&lt;/a&gt;, what are your thoughts?&lt;/p&gt;</comment>
                            <comment id="3066974" author="tess.avitabile" created="Mon, 4 May 2020 13:51:30 +0000"  >&lt;p&gt;Thanks, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=dianna.hohensee&quot; class=&quot;user-hover&quot; rel=&quot;dianna.hohensee&quot;&gt;dianna.hohensee&lt;/a&gt;!&lt;/p&gt;

&lt;p&gt;Yes, the lastApplied optime is advanced through on-commit handling, so it can lag behind what is written in the oplog.&lt;/p&gt;

&lt;p&gt;Since this is expected, we should probably either (1) Remove the log line, or (2) Update the lastApplied when we update the lastDurable. I think (2) is preferable, since having lastApplied behind lastDurable will lead us to log surprising heartbeat messages, and I&apos;m not sure if we have any other assumptions that lastApplied should be ahead of lastDurable.&lt;/p&gt;</comment>
                            <comment id="3066919" author="dianna.hohensee" created="Mon, 4 May 2020 13:32:11 +0000"  >&lt;p&gt;It sounds possible to me that the durable timestamp can move ahead of the last applied timestamp. I am not familiar with how the applied timestamp is updated &#8211; presumable some kind of on commit handling? --, but the durable timestamp is being updated independently in mode PRIMARY by asynchronous replicate before journaling logic.&lt;/p&gt;

&lt;p&gt;Flushing the journal will first &lt;a href=&quot;https://github.com/mongodb/mongo/blob/01583bbf5e0884c94dcb2545be47cc3104ccbd11/src/mongo/db/repl/replication_coordinator_external_state_impl.cpp#L1054-L1059&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;fetch the OpTimeAndWallTime of the oplog entry associated with the WT all_durable timestamp&lt;/a&gt; (the no oplog holes point in-memory). Then do the journal flush. Then use the previously fetched OpTimeAndWallTime to &lt;a href=&quot;https://github.com/mongodb/mongo/blob/01583bbf5e0884c94dcb2545be47cc3104ccbd11/src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp#L309&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;update the repl durable timestamp&lt;/a&gt;. The durable timestamp updates for journaling are dependent only on the WT all_durable timestamp.&lt;/p&gt;

&lt;p&gt;In mode SECONDARY, the oplog applier thread &lt;a href=&quot;https://github.com/mongodb/mongo/blob/01583bbf5e0884c94dcb2545be47cc3104ccbd11/src/mongo/db/repl/oplog_applier_impl.cpp#L734&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;explicitly triggers journal flushes, but moves on without waiting&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="3066849" author="tess.avitabile" created="Mon, 4 May 2020 12:31:26 +0000"  >&lt;p&gt;That does sound concerning. It would be great to see logs with increased log level.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=dianna.hohensee&quot; class=&quot;user-hover&quot; rel=&quot;dianna.hohensee&quot;&gt;dianna.hohensee&lt;/a&gt;, do you think this could be related to the Replicate Before Journaling project?&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=matthew.russotto&quot; class=&quot;user-hover&quot; rel=&quot;matthew.russotto&quot;&gt;matthew.russotto&lt;/a&gt;, we still expect it to be an error for the lastDurable optime to advance beyond the lastApplied optime, is that right?&lt;/p&gt;</comment>
                            <comment id="3065250" author="thomas.schubert" created="Fri, 1 May 2020 20:40:34 +0000"  >&lt;p&gt;Here&apos;s a sample log line it happens every few seconds:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{&quot;t&quot;:{&quot;$date&quot;:&quot;2020-05-01T19:38:57.624Z&quot;},&quot;s&quot;:&quot;I&quot;,&quot;c&quot;:&quot;REPL&quot;,&quot;id&quot;:21218,&quot;ctx&quot;:&quot;WTJournalFlusher&quot;,&quot;msg&quot;:&quot;Durable progress is ahead of the applied progress. This is likely due to a rollback&quot;,&quot;attr&quot;:{&quot;durableOpTime&quot;:{&quot;ts&quot;:{&quot;_timestamp&quot;:{&quot;t&quot;:1588361937,&quot;i&quot;:1512}},&quot;t&quot;:56},&quot;lastAppliedOpTime&quot;:{&quot;ts&quot;:{&quot;_timestamp&quot;:{&quot;t&quot;:1588361937,&quot;i&quot;:1490}},&quot;t&quot;:56},&quot;memberId&quot;:&quot;MemberId(2)&quot;,&quot;hostAndPort&quot;:&quot;logkeeperdb-88.10gen-mci.4085.mongodbdns.com:27017&quot;,&quot;lastDurableOpTime&quot;:{&quot;ts&quot;:{&quot;_timestamp&quot;:{&quot;t&quot;:1588361937,&quot;i&quot;:1393}},&quot;t&quot;:56}}} &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="1348668">SERVER-48149</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1413502">SERVER-49695</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1475010">SERVER-50949</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is depended on by">
                                        <issuelink>
            <issuekey id="2554400">SERVER-85600</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10520">
                    <name>Problem/Incident</name>
                                            <outwardlinks description="causes">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="1535935">SERVER-52661</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2522206">SERVER-84082</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="1339455">SERVER-47941</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>29.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25128"><![CDATA[Replication]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="18953"><![CDATA[v4.4]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 4 May 2020 12:31:26 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        8 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/SERVER-48149'>SERVER-48149</a></s>, <s><a href='https://jira.mongodb.org/browse/SERVER-49695'>SERVER-49695</a></s>, <s><a href='https://jira.mongodb.org/browse/SERVER-50949'>SERVER-50949</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-3489</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>brad.cater@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            8 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>33.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>daniel.gottlieb@mongodb.com</customfieldvalue>
            <customfieldvalue>dianna.hohensee@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>judah.schvimer@mongodb.com</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
            <customfieldvalue>kishore.devireddy@mongodb.com</customfieldvalue>
            <customfieldvalue>tess.avitabile@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxixxj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr4kzj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_22250" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Special Downgrade Instructions Required</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="23343"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="4140">Repl 2020-08-24</customfieldvalue>
    <customfieldvalue id="4215">Execution Team 2020-10-19</customfieldvalue>
    <customfieldvalue id="7967">Repl 2023-12-11</customfieldvalue>
    <customfieldvalue id="7968">Repl 2023-12-25</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxik6v:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>