<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:51:04 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-39112] Primary drain mode can be unnecessarily slow</title>
                <link>https://jira.mongodb.org/browse/SERVER-39112</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;After a replica set node wins an election and transitions to PRIMARY state, it enters drain mode. In this mode, it will apply any oplog operations that were still left in its buffer from its time as a secondary. While in drain mode, a node is in PRIMARY state but cannot yet accept writes i.e. it will &lt;a href=&quot;https://github.com/mongodb/mongo/blob/bfb0aec447f39633694dcc4418f2a5a5a167937e/src/mongo/db/repl/replication_coordinator_impl.cpp#L2176-L2177&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;report&lt;/a&gt; &lt;tt&gt;isMaster:false&lt;/tt&gt;. When the drain process has completed, the ReplicationCoordinator will be signaled by the oplog application logic in SyncTail. In the case that there are no operations to apply in drain mode, though, the newly elected primary should be able to complete drain mode immediately and begin accepting writes. This process may take up to a second or more, though, because of this &lt;a href=&quot;https://github.com/mongodb/mongo/blob/bfb0aec447f39633694dcc4418f2a5a5a167937e/src/mongo/db/repl/sync_tail.cpp#L858-L860&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;hard coded 1 second timeout&lt;/a&gt; in the oplog application loop. This is wasted downtime where the primary could be accepting writes but is waiting for this timeout to trigger. This limits how quickly a node can step up and begin accepting writes. We should consider making this timeout configurable via an external parameter or hard-coding it at something less i.e. 100 milliseconds. Perhaps the ReplicationCoordinator could also signal the oplog application loop when it transitions to PRIMARY, letting it know it can check right away if drain mode can complete.&lt;/p&gt;</description>
                <environment></environment>
        <key id="675662">SERVER-39112</key>
            <summary>Primary drain mode can be unnecessarily slow</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="matthew.russotto@mongodb.com">Matthew Russotto</assignee>
                                    <reporter username="william.schultz@mongodb.com">William Schultz</reporter>
                        <labels>
                    </labels>
                <created>Mon, 21 Jan 2019 20:29:06 +0000</created>
                <updated>Sun, 29 Oct 2023 22:24:56 +0000</updated>
                            <resolved>Thu, 27 Feb 2020 17:04:31 +0000</resolved>
                                    <version>3.4.18</version>
                    <version>3.6.10</version>
                    <version>4.0.5</version>
                    <version>4.1.7</version>
                                    <fixVersion>4.2.4</fixVersion>
                    <fixVersion>4.3.4</fixVersion>
                    <fixVersion>4.0.17</fixVersion>
                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>21</watches>
                                                                                                                <comments>
                            <comment id="2914892" author="xgen-internal-githook" created="Thu, 27 Feb 2020 19:16:19 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;username&apos;: &apos;mtrussotto&apos;, &apos;name&apos;: &apos;Matthew Russotto&apos;, &apos;email&apos;: &apos;matthew.russotto@10gen.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-39112&quot; title=&quot;Primary drain mode can be unnecessarily slow&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-39112&quot;&gt;&lt;del&gt;SERVER-39112&lt;/del&gt;&lt;/a&gt; Remove 1-second delays when starting and stopping OplogApplier. Speeds up transition from primary drain mode to a writeable primary.&lt;br/&gt;
Branch: v4.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/c278fcebdc161bc589fd4869a51019cfa1cdaf8c&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/c278fcebdc161bc589fd4869a51019cfa1cdaf8c&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2914634" author="xgen-internal-githook" created="Thu, 27 Feb 2020 17:02:14 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Matthew Russotto&apos;, &apos;username&apos;: &apos;mtrussotto&apos;, &apos;email&apos;: &apos;matthew.russotto@10gen.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-39112&quot; title=&quot;Primary drain mode can be unnecessarily slow&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-39112&quot;&gt;&lt;del&gt;SERVER-39112&lt;/del&gt;&lt;/a&gt; Remove 1-second delays when starting and stopping OplogApplier. Speeds up transition from primary drain mode to a writeable primary.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/869dbc8bd522338c2ce25e679c4772952c1f1fa6&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/869dbc8bd522338c2ce25e679c4772952c1f1fa6&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2914577" author="xgen-internal-githook" created="Thu, 27 Feb 2020 16:43:54 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Matthew Russotto&apos;, &apos;username&apos;: &apos;mtrussotto&apos;, &apos;email&apos;: &apos;matthew.russotto@10gen.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-39112&quot; title=&quot;Primary drain mode can be unnecessarily slow&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-39112&quot;&gt;&lt;del&gt;SERVER-39112&lt;/del&gt;&lt;/a&gt; Remove 1-second delays when starting and stopping OplogApplier. Speeds up transition from primary drain mode to a writeable primary.&lt;br/&gt;
Branch: v4.2&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/a9cf638d70ed4377d4d1426c1d7fec1464417a08&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/a9cf638d70ed4377d4d1426c1d7fec1464417a08&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2785526" author="matthew.russotto" created="Wed, 5 Feb 2020 19:27:34 +0000"  >&lt;p&gt;Since the OplogBuffer itself allows only one producer, a possible implementation might be a drain mode for the oplog buffer.  We could enter oplog buffer drain mode from the producer; this would interrupt any waits and ignore future waits.  Then when drain was complete, or when we restarted the oplog fetcher, we could exit the oplog buffer drain mode.   This avoids adding any new synchronization between the fetcher and the batcher.&lt;/p&gt;</comment>
                            <comment id="2785465" author="matthew.russotto" created="Wed, 5 Feb 2020 18:59:07 +0000"  >&lt;p&gt;A solution gets a bit tricky.  What we want is, when we&apos;re in drain mode, and the oplog fetcher is not running, for the ReplBatcher to stop waiting and to not wait again until we drop out of drain mode.  Drain mode is not currently synchronized with the ReplBatcher; we might be able to do that but we&apos;d have to be very careful with the lock ordering.   We cannot have the OplogBufferBlockingQueue not wait whenever the oplog fetcher is not running, as the ReplBatcher runs when we are not SECONDARY, so we depend on that 1 second wait to avoid a full cpu-burning spin.&lt;/p&gt;</comment>
                            <comment id="2784976" author="matthew.russotto" created="Wed, 5 Feb 2020 16:15:24 +0000"  >&lt;p&gt;There are two waits in sync_tail,&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/ee21b1376e7fb6502db30652ebdf0a65ac876de4/src/mongo/db/repl/sync_tail.cpp#L617&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/ee21b1376e7fb6502db30652ebdf0a65ac876de4/src/mongo/db/repl/sync_tail.cpp#L617&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;This is waiting for a batch to be available from the ReplBatcher.  Since &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42219&quot; title=&quot;Oplog buffer not always empty when primary exits drain mode&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42219&quot;&gt;&lt;del&gt;SERVER-42219&lt;/del&gt;&lt;/a&gt;, I believe this no longer causes a delay because the condition variable will be pinged and _ops.termWhenExhausted() set when we enter drain mode.&lt;/p&gt;

&lt;p&gt;The other is this one&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/ee21b1376e7fb6502db30652ebdf0a65ac876de4/src/mongo/db/repl/sync_tail.cpp#L681&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/ee21b1376e7fb6502db30652ebdf0a65ac876de4/src/mongo/db/repl/sync_tail.cpp#L681&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;This is waiting for a batch to become available from the oplog fetcher.  This is definitely causing a delay.  The oplog fetcher knows when it will no longer be adding anything to the queue (this is when it calls the finalizer callback); however, the underlying queue has no way of interrupting a wait.  This may be a good reason to implement &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40049&quot; title=&quot;Make OplogBuffer use ProducerConsumerQueue instead of BlockingQueue&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40049&quot;&gt;&lt;del&gt;SERVER-40049&lt;/del&gt;&lt;/a&gt;, which would switch to a more capable queue. &lt;/p&gt;

&lt;p&gt;This only explains up to 1 second of delay, but I believe the multi-second delays seen in the linked help ticket may be due to a DNS infrastructure problem.&lt;/p&gt;

&lt;p&gt;(edit, fixed line number of second wait)&lt;/p&gt;</comment>
                            <comment id="2782734" author="matthew.russotto" created="Tue, 4 Feb 2020 16:22:00 +0000"  >&lt;p&gt;The message &quot;Oplog buffer has been drained&quot; (&lt;a href=&quot;https://github.com/mongodb/mongo/blob/b5134bcb006e74e2102de6e171a28931f84ea12f/src/mongo/db/repl/oplog_batcher.cpp#L322&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/b5134bcb006e74e2102de6e171a28931f84ea12f/src/mongo/db/repl/oplog_batcher.cpp#L322&lt;/a&gt;)  means we&apos;re done; we could ping the condition variable that terminates the wait there.   However, this only saves 10-20ms, so it&apos;s not the main issue.&lt;/p&gt;

&lt;p&gt;Actually we&apos;ve already exited the wait there, so not helpful at all.&lt;/p&gt;</comment>
                            <comment id="2634417" author="william.schultz" created="Mon, 16 Dec 2019 16:44:24 +0000"  >&lt;p&gt;I ran another quick test on my local workstation after the changes from this ticket and compared the performance of drain mode when there are no ops to apply on master vs v4.2, using this test (&lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/240522/240522_drain.js&quot; title=&quot;drain.js attached to SERVER-39112&quot;&gt;drain.js&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;). I measured the speed of drain mode when the second primary stepped up:&lt;/p&gt;

&lt;p&gt;&lt;b&gt;master&lt;/b&gt;&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;// 226ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:drain] 2019-12-16T11:27:32.429-0500 d20021| 2019-12-16T11:27:32.428-0500 I  REPL     [ReplCoord-1] transition to PRIMARY from SECONDARY&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:drain] 2019-12-16T11:27:32.569-0500 [jsTest] waiting for new primary.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:drain] 2019-12-16T11:27:32.652-0500 d20021| 2019-12-16T11:27:32.652-0500 I  REPL     [ReplBatcher] Oplog buffer has been drained in term 2&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:drain] 2019-12-16T11:27:32.654-0500 d20021| 2019-12-16T11:27:32.654-0500 I  REPL     [OplogApplier-0] transition to primary complete; database writes are now permitted&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;// 212ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:drain] 2019-12-16T11:27:40.502-0500 d20021| 2019-12-16T11:27:40.502-0500 I  REPL     [ReplCoord-0] transition to PRIMARY from SECONDARY&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:drain] 2019-12-16T11:27:40.643-0500 [jsTest] waiting for new primary.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:drain] 2019-12-16T11:27:40.709-0500 d20021| 2019-12-16T11:27:40.709-0500 I  REPL     [ReplBatcher] Oplog buffer has been drained in term 2&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:drain] 2019-12-16T11:27:40.714-0500 d20021| 2019-12-16T11:27:40.714-0500 I  REPL     [OplogApplier-0] transition to primary complete; database writes are now permitted&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;// 229ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:drain] 2019-12-16T11:27:49.425-0500 d20021| 2019-12-16T11:27:49.425-0500 I  REPL     [ReplCoord-2] transition to PRIMARY from SECONDARY&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:drain] 2019-12-16T11:27:49.566-0500 [jsTest] waiting for new primary.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:drain] 2019-12-16T11:27:49.648-0500 d20021| 2019-12-16T11:27:49.648-0500 I  REPL     [ReplBatcher] Oplog buffer has been drained in term 2&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:drain] 2019-12-16T11:27:49.654-0500 d20021| 2019-12-16T11:27:49.654-0500 I  REPL     [OplogApplier-0] transition to primary complete; database writes are now permitted&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;&lt;b&gt;v4.2&lt;/b&gt;&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;// 779ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:drain] 2019-12-16T11:34:09.804-0500 d20021| 2019-12-16T11:34:09.803-0500 I  REPL     [replexec-0] transition to PRIMARY from SECONDARY&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:drain] 2019-12-16T11:34:10.010-0500 [jsTest] waiting for new primary.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:drain] 2019-12-16T11:34:10.579-0500 d20021| 2019-12-16T11:34:10.578-0500 I  REPL     [ReplBatcher] Oplog buffer has been drained in term 2                                                  &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:drain] 2019-12-16T11:34:10.582-0500 d20021| 2019-12-16T11:34:10.582-0500 I  REPL     [rsSync-0] transition to primary complete; database writes are now permitted&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;// 485ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:drain] 2019-12-16T11:34:24.247-0500 d20021| 2019-12-16T11:34:24.247-0500 I  REPL     [replexec-0] transition to PRIMARY from SECONDARY&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:drain] 2019-12-16T11:34:24.454-0500 [jsTest] waiting for new primary.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:drain] 2019-12-16T11:34:24.728-0500 d20021| 2019-12-16T11:34:24.728-0500 I  REPL     [ReplBatcher] Oplog buffer has been drained in term 2&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:drain] 2019-12-16T11:34:24.733-0500 d20021| 2019-12-16T11:34:24.732-0500 I  REPL     [rsSync-0] transition to primary complete; database writes are now permitted&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;// 831ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:drain] 2019-12-16T11:34:38.381-0500 d20021| 2019-12-16T11:34:38.381-0500 I  REPL     [replexec-2] transition to PRIMARY from SECONDARY&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:drain] 2019-12-16T11:34:38.587-0500 [jsTest] waiting for new primary.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:drain] 2019-12-16T11:34:39.207-0500 d20021| 2019-12-16T11:34:39.206-0500 I  REPL     [ReplBatcher] Oplog buffer has been drained in term 2&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:drain] 2019-12-16T11:34:39.212-0500 d20021| 2019-12-16T11:34:39.212-0500 I  REPL     [rsSync-0] transition to primary complete; database writes are now permitted&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;On master the mean over 3 runs was 222ms and on v4.2 the mean over 3 runs was 698ms. This definitely seems to have improved but it&apos;s not exactly clear if it could still be faster. I would also want to test it on a different machine.&lt;/p&gt;
</comment>
                            <comment id="2477372" author="siyuan.zhou@10gen.com" created="Fri, 11 Oct 2019 01:38:32 +0000"  >&lt;p&gt;Sounds good! Thank you.&lt;/p&gt;</comment>
                            <comment id="2476366" author="william.schultz" created="Thu, 10 Oct 2019 19:29:22 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=siyuan.zhou&quot; class=&quot;user-hover&quot; rel=&quot;siyuan.zhou&quot;&gt;siyuan.zhou&lt;/a&gt; I saw your analysis on &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42219&quot; title=&quot;Oplog buffer not always empty when primary exits drain mode&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42219&quot;&gt;&lt;del&gt;SERVER-42219&lt;/del&gt;&lt;/a&gt;. I would like to take one more look at this to see how the performance changed. I think I will be able to get to this as a part of the Faster Local Testing (PM-1360) project. For now I will leave this ticket open if that is OK with you.&lt;/p&gt;</comment>
                            <comment id="2474700" author="siyuan.zhou@10gen.com" created="Wed, 9 Oct 2019 21:07:27 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=william.schultz&quot; class=&quot;user-hover&quot; rel=&quot;william.schultz&quot;&gt;william.schultz&lt;/a&gt;, the hard-coded one second waiting can be signaled when the oplog buffer is drained in&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42219&quot; title=&quot;Oplog buffer not always empty when primary exits drain mode&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42219&quot;&gt;&lt;del&gt;SERVER-42219&lt;/del&gt;&lt;/a&gt;.&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42219&quot; title=&quot;Oplog buffer not always empty when primary exits drain mode&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42219&quot;&gt;&lt;del&gt;SERVER-42219&lt;/del&gt;&lt;/a&gt; also showed the failover time is shorten. Do you think that covers your original concern in this ticket?&lt;/p&gt;</comment>
                            <comment id="2137263" author="siyuan.zhou@10gen.com" created="Mon, 4 Feb 2019 20:32:32 +0000"  >&lt;p&gt;I don&apos;t think the transition from secondary to primary will leave any ops in the buffer, since both the states of bgsync and applier are set in &lt;tt&gt;_enterDrainMode_inlock&lt;/tt&gt; atomically, where bgsync changes its state to &lt;tt&gt;ProducerState::Stopped&lt;/tt&gt;. Fetcher is then canceled. On receiving a new batch, &lt;tt&gt;BackgroundSync::_enqueueDocuments&lt;/tt&gt; checks the state again.&lt;/p&gt;

&lt;p&gt;It&apos;s possible to start bgsync with ops in the buffer though. As explained by &lt;a href=&quot;https://github.com/mongodb/mongo/blob/87ca277a86cba6fb7e129e341e8785eada8a1589/src/mongo/db/repl/bgsync.cpp#L738-L742&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this comment&lt;/a&gt;, bgsync will see ops left in the buffer if the node steps down during drain mode.&lt;/p&gt;

&lt;p&gt;Although I don&apos;t think there&apos;s a correctness concern here, I do think it&apos;s worthwhile to improve the signaling. The &lt;a href=&quot;https://github.com/mongodb/mongo/blob/82f36212ab8be35afebc66b92605071d9e4bbc5c/src/mongo/db/repl/replication_coordinator.h#L446-L492&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;state transition of bgsync and applier&lt;/a&gt; needs to be considered and the locking rules around &lt;a href=&quot;https://github.com/mongodb/mongo/blob/d764e1b6346726fc2e9259ba294d029d531ff38e/src/mongo/db/repl/bgsync.h#L233&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;bgsync&apos;s mutex&lt;/a&gt; and coordinator&apos;s mutex need to be followed.&lt;/p&gt;</comment>
                            <comment id="2124700" author="judah.schvimer" created="Wed, 23 Jan 2019 20:38:09 +0000"  >&lt;p&gt;The bug you describe around operations sticking around in the buffer seems very plausible. Additionally, do we make decisions based on &lt;tt&gt;lastFetched&lt;/tt&gt; assuming that those will always be applied?&lt;/p&gt;</comment>
                            <comment id="2124614" author="william.schultz" created="Wed, 23 Jan 2019 19:28:40 +0000"  >&lt;p&gt;From a correctness perspective it doesn&apos;t seem harmful to throw away some ops that were in your buffer during drain mode. You would simply become primary with some prefix of the log entries you could have had. Like you said, this may mitigate the benefits of drain and/or catchup mode, though.&lt;/p&gt;

&lt;p&gt;One issue I could think of is if those operations stick around in the buffer while you are primary and remain there until you become secondary again. That could be problematic if the applier starts pulling ops out of the queue in FIFO order after you&apos;ve written down a bunch of oplog entries as a primary. Perhaps we clean out this buffer at some point, though, between these state transitions.&lt;/p&gt;</comment>
                            <comment id="2123546" author="judah.schvimer" created="Tue, 22 Jan 2019 22:34:46 +0000"  >&lt;blockquote&gt;
&lt;p&gt;If we end drain mode immediately when the buffer appears to be empty, we might possibly be abandoning some operations that are still pending that the network thread could have captured.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;It sounds like in this case we have an existing bug since waiting 1 second doesn&apos;t provide any assurances beyond exiting drain mode immediately. If we currently abandon some operations, what&apos;s the harm? Unnecessary rollbacks?&lt;/p&gt;</comment>
                            <comment id="2123347" author="milkie" created="Tue, 22 Jan 2019 21:08:34 +0000"  >&lt;p&gt;Yes, something like that would fix this.&#160; Synchronization between the producer and the applier used to be tricky (some bugs happened in this area), but I am hopeful the code&apos;s in a better state today to make this happen.&lt;/p&gt;</comment>
                            <comment id="2123211" author="william.schultz" created="Tue, 22 Jan 2019 19:39:21 +0000"  >&lt;p&gt;Perhaps the producer and applier could coordinate to signal the completion of drain mode more efficiently. To &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=milkie&quot; class=&quot;user-hover&quot; rel=&quot;milkie&quot;&gt;milkie&lt;/a&gt;&apos;s point about waiting for operations captured by the network thread, it seems that after entering drain mode we want to ensure that (1) the producer has stopped and is not going to fetch any more operations, and, once (1) holds, that the applier has no more ops to apply in its buffer. Perhaps the producer, once it is shut down and knows it will not fetch any more operations, could signal the applier to do one last check to drain its operations buffer. If the buffer is empty, the applier can signal drain completion immediately. Otherwise, it can drain the remaining ops and then signal. &lt;/p&gt;</comment>
                            <comment id="2123155" author="milkie" created="Tue, 22 Jan 2019 19:08:51 +0000"  >&lt;p&gt;I went back and checked the original code to see why it was written this way, and it turns out we&apos;ve lost an important comment that accompanied the original code.&#160; The reason why we need the delay here is because the network receiving thread and the applier thread are running asynchronously, and we wanted to wait to ensure that all the operations that could be captured by the network thread are processed in drain mode.&#160; If we end drain mode immediately when the buffer appears to be empty, we might possibly be abandoning some operations that are still pending that the network thread could have captured.&lt;/p&gt;</comment>
                            <comment id="2123149" author="milkie" created="Tue, 22 Jan 2019 19:05:28 +0000"  >&lt;p&gt;I don&apos;t see why we can&apos;t just call signalDrainComplete() immediately when the buffer is first known to be empty.&#160; Then we can keep the 1 second timeout the same for the other less important checks.&lt;/p&gt;

&lt;p&gt;You&apos;d have to change the batcher to be able to support this behavior, but shouldn&apos;t be too hard.&lt;/p&gt;</comment>
                            <comment id="2123139" author="alyson.cabral" created="Tue, 22 Jan 2019 18:58:42 +0000"  >&lt;p&gt;This proposed change seems very impactful. Is there a tradeoff to making the timeout less? Even 100MS wasted seems high.&lt;/p&gt;</comment>
                            <comment id="2122118" author="william.schultz" created="Mon, 21 Jan 2019 20:30:49 +0000"  >&lt;p&gt;When running the simple demo script  &lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/206804/206804_replset_startup.js&quot; title=&quot;replset_startup.js attached to SERVER-39112&quot;&gt;replset_startup.js&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;, we can see that the interval between &lt;tt&gt;transition to PRIMARY&lt;/tt&gt; and &lt;tt&gt;transition to primary complete; database writes are now permitted&lt;/tt&gt; messages are well over 1 second:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;d20020| 2019-01-21T15:12:47.624-0500 I REPL     [replexec-0] transition to PRIMARY from SECONDARY&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;d20020| 2019-01-21T15:12:49.634-0500 I REPL     [rsSync-0] transition to primary complete; database writes are now permitted&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;d20021| 2019-01-21T15:12:50.587-0500 I REPL     [rsSync-0] transition to SECONDARY from RECOVERING&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;d20020| 2019-01-21T15:12:52.146-0500 I REPL     [conn1] transition to SECONDARY from PRIMARY&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;d20020| 2019-01-21T15:12:52.147-0500 I REPL     [conn1] Handing off election to williams-ubuntu:20021&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;d20021| 2019-01-21T15:12:52.214-0500 I REPL     [replexec-1] transition to PRIMARY from SECONDARY&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;d20021| 2019-01-21T15:12:53.620-0500 I REPL     [rsSync-0] transition to primary complete; database writes are now permitted&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;If we alter the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/bfb0aec447f39633694dcc4418f2a5a5a167937e/src/mongo/db/repl/sync_tail.cpp#L860&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;aforementioned timeout&lt;/a&gt; to 100 milliseconds, the intervals shrink to around ~100 milliseconds or less. &lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;d20020| 2019-01-21T14:57:26.679-0500 I REPL     [replexec-0] transition to PRIMARY from SECONDARY&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;d20020| 2019-01-21T14:57:26.782-0500 I REPL     [rsSync-0] transition to primary complete; database writes are now permitted&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;d20021| 2019-01-21T14:57:27.630-0500 I REPL     [rsSync-0] transition to SECONDARY from RECOVERING&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;d20020| 2019-01-21T14:57:29.211-0500 I REPL     [conn1] transition to SECONDARY from PRIMARY&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;d20020| 2019-01-21T14:57:29.211-0500 I REPL     [conn1] Handing off election to williams-ubuntu:20021&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;d20021| 2019-01-21T14:57:29.294-0500 I REPL     [replexec-1] transition to PRIMARY from SECONDARY&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;d20021| 2019-01-21T14:57:29.336-0500 I REPL     [rsSync-0] transition to primary complete; database writes are now permitted&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;This is something I noticed in testing, and was confused by why the primary step up process appeared so slow. I think that making this process faster could be beneficial for our test infrastructure i.e. making setup times in ReplSetTest quicker, and may also be important for production systems that are sensitive to lost write availability. The election handoff project, for example, made planned failovers significantly faster. I think that removing an additional, wasted second from that process would be valuable. &lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="337357">SERVER-27342</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="855998">SERVER-42219</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="240522" name="drain.js" size="506" author="william.schultz@mongodb.com" created="Mon, 16 Dec 2019 16:43:22 +0000"/>
                            <attachment id="206804" name="replset_startup.js" size="369" author="william.schultz@mongodb.com" created="Mon, 21 Jan 2019 20:29:39 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>21.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16775"><![CDATA[v4.2]]></customfieldvalue>
    <customfieldvalue key="15640"><![CDATA[v4.0]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 22 Jan 2019 18:58:42 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 49 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 49 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>alyson.cabral@mongodb.com</customfieldvalue>
            <customfieldvalue>milkie@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>judah.schvimer@mongodb.com</customfieldvalue>
            <customfieldvalue>matthew.russotto@mongodb.com</customfieldvalue>
            <customfieldvalue>siyuan.zhou@mongodb.com</customfieldvalue>
            <customfieldvalue>william.schultz@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hujukf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hwbvhj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3575">Repl 2020-02-10</customfieldvalue>
    <customfieldvalue id="3576">Repl 2020-02-24</customfieldvalue>
    <customfieldvalue id="3577">Repl 2020-03-09</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hujgtr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>