<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:04:55 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-44068] Investigate the slowest sections of RollbackTest and remove any wasted downtime</title>
                <link>https://jira.mongodb.org/browse/SERVER-44068</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We should investigate the slowest parts of the RollbackTest fixture to identify and remove any areas of wasted downtime. Wasted downtime would be any time spent not doing useful, required work. For example, waiting for data to propagate over the network or waiting for an arbitrary timeout to fire in the server e.g. a heartbeat interval or election timeout.&lt;/p&gt;

&lt;p&gt;In many of our tests that use the rollback test fixture, the duration of the rollback itself is often only a few hundred milliseconds. In theory, our test fixtures should be able to run a rollback test sequence in a time not much longer than it takes to complete a rollback, setting aside required setup and teardown time. We should investigate to see if this is feasible with our current testing infrastructure.&lt;/p&gt;</description>
                <environment></environment>
        <key id="972197">SERVER-44068</key>
            <summary>Investigate the slowest sections of RollbackTest and remove any wasted downtime</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="4" iconUrl="https://jira.mongodb.org/images/icons/priorities/minor.svg">Minor - P4</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="william.schultz@mongodb.com">William Schultz</assignee>
                                    <reporter username="william.schultz@mongodb.com">William Schultz</reporter>
                        <labels>
                            <label>former-quick-wins</label>
                    </labels>
                <created>Thu, 17 Oct 2019 17:22:13 +0000</created>
                <updated>Sun, 29 Oct 2023 22:16:01 +0000</updated>
                            <resolved>Mon, 27 Apr 2020 22:23:08 +0000</resolved>
                                                    <fixVersion>4.7.0</fixVersion>
                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>2</watches>
                                                                                                                <comments>
                            <comment id="3057637" author="william.schultz" created="Mon, 27 Apr 2020 22:33:15 +0000"  >&lt;p&gt;I think we may want to backport the &apos;restartHeartbeats&apos; command (i.e. this &lt;a href=&quot;https://github.com/mongodb/mongo/commit/a697d23b924a47084501caee38365970f89852d3&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;commit&lt;/a&gt;) to 4.4 so that we don&apos;t run in to multiversion failures when we re-enable multiversion suites, which are currently disabled (see &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-47083&quot; title=&quot;Disable multiversion passthroughs until FCV is bumped to 4.6&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-47083&quot;&gt;&lt;del&gt;SERVER-47083&lt;/del&gt;&lt;/a&gt;). &lt;/p&gt;</comment>
                            <comment id="3057533" author="william.schultz" created="Mon, 27 Apr 2020 21:09:25 +0000"  >&lt;p&gt;After the final changes have been pushed, we can see the overall improvements from this ticket, by measuring the durations of transitions inside the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/dc116507128f70cc2399bf6656764313aa7d7333/jstests/replsets/rollback_test_control.js&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;rollback_test_control.js&lt;/a&gt; test on my local workstation with &lt;tt&gt;--opt=on --dbg=on&lt;/tt&gt; build flags:&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Before&lt;/b&gt;&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:00.516+0000 [RollbackTest] RollbackTest transition to kRollbackOps took 14 ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:01.349+0000 [RollbackTest] RollbackTest transition to kSyncSourceOpsBeforeRollback took 835 ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:01.350+0000 [RollbackTest] RollbackTest transition to kSyncSourceOpsDuringRollback took 1 ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:04.197+0000 [RollbackTest] RollbackTest transition to kSteadyStateOps took 2847 ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:04.214+0000 [RollbackTest] RollbackTest transition to kStopped took 16 ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;&lt;b&gt;After&lt;/b&gt;&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-27T17:06:32.471-0400 [RollbackTest] RollbackTest transition to kRollbackOps took 19 ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-27T17:06:32.572-0400 [RollbackTest] RollbackTest transition to kSyncSourceOpsBeforeRollback took 101 ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-27T17:06:32.574-0400 [RollbackTest] RollbackTest transition to kSyncSourceOpsDuringRollback took 2 ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-27T17:06:33.448-0400 [RollbackTest] RollbackTest transition to kSteadyStateOps took 874 ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-27T17:06:33.465-0400 [RollbackTest] RollbackTest transition to kStopped took 17 ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;This represents an approximate reduction of 700ms for the &lt;tt&gt;kSyncSourceOpsBeforeRollback&lt;/tt&gt; transition and 1973ms for the &lt;tt&gt;kSteadyStateOps&lt;/tt&gt; transition. &lt;/p&gt;

&lt;p&gt;To understand these numbers a bit more accurately, we can also look at the before/after distributions of durations for the transitions we improved. the numbers below are taken from 20 repeated runs of rollback_test_control.js on my local workstation, before and after the changes.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;kSyncSourceOpsBeforeRollback, Before&lt;/b&gt;  &lt;br/&gt;
Median: 842.5ms&lt;br/&gt;
SD: 305.3ms&lt;br/&gt;
95 pctile: 866.15ms&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;257703_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/257703/257703_before_stats_kSyncSourceOpsBeforeRollback.png&quot; title=&quot;before_stats_kSyncSourceOpsBeforeRollback.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;257703&quot; file-preview-title=&quot;before_stats_kSyncSourceOpsBeforeRollback.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/257703/_thumb_257703.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;&lt;b&gt;kSyncSourceOpsBeforeRollback, After&lt;/b&gt;  &lt;br/&gt;
Median: 65.5ms&lt;br/&gt;
SD: 25.1ms&lt;br/&gt;
95 pctile: 108.8ms&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;257704_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/257704/257704_after_stats_kSyncSourceOpsBeforeRollback.png&quot; title=&quot;after_stats_kSyncSourceOpsBeforeRollback.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;257704&quot; file-preview-title=&quot;after_stats_kSyncSourceOpsBeforeRollback.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/257704/_thumb_257704.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;&lt;b&gt;kSteadyStateOps, Before&lt;/b&gt;&lt;br/&gt;
Median: 2975.5ms&lt;br/&gt;
SD: 369.5ms&lt;br/&gt;
95 pctile: 3590.9ms&lt;br/&gt;
 &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;257701_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/257701/257701_before_stats_kSteadyStateOps.png&quot; title=&quot;before_stats_kSteadyStateOps.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;257701&quot; file-preview-title=&quot;before_stats_kSteadyStateOps.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/257701/_thumb_257701.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;&lt;b&gt;kSteadyStateOps, After&lt;/b&gt;&lt;br/&gt;
Median: 877.5ms&lt;br/&gt;
SD: 103.4ms&lt;br/&gt;
95 pctile: 1143.45ms&lt;br/&gt;
 &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;257702_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/257702/257702_after_stats_kSteadyStateOps.png&quot; title=&quot;after_stats_kSteadyStateOps.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;257702&quot; file-preview-title=&quot;after_stats_kSteadyStateOps.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/257702/_thumb_257702.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;
</comment>
                            <comment id="3057319" author="xgen-internal-githook" created="Mon, 27 Apr 2020 19:32:49 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;William Schultz&apos;, &apos;email&apos;: &apos;william.schultz@mongodb.com&apos;, &apos;username&apos;: &apos;will62794&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-44068&quot; title=&quot;Investigate the slowest sections of RollbackTest and remove any wasted downtime&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-44068&quot;&gt;&lt;del&gt;SERVER-44068&lt;/del&gt;&lt;/a&gt; Speed up transition to &apos;kSteadyStateOps&apos; in RollbackTest&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/a30cef00b9e3ccfe60a4dce7f9f2c7cbaf1e68b7&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/a30cef00b9e3ccfe60a4dce7f9f2c7cbaf1e68b7&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3039802" author="xgen-internal-githook" created="Tue, 14 Apr 2020 21:51:51 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;William Schultz&apos;, &apos;email&apos;: &apos;william.schultz@mongodb.com&apos;, &apos;username&apos;: &apos;will62794&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-44068&quot; title=&quot;Investigate the slowest sections of RollbackTest and remove any wasted downtime&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-44068&quot;&gt;&lt;del&gt;SERVER-44068&lt;/del&gt;&lt;/a&gt; Speed up transition to &apos;kSyncSourceOpsBeforeRollback&apos; in RollbackTest&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/d0b9ca3aa8015954b9d0fa4f2f14c142c32625f5&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/d0b9ca3aa8015954b9d0fa4f2f14c142c32625f5&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3039801" author="xgen-internal-githook" created="Tue, 14 Apr 2020 21:51:49 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;William Schultz&apos;, &apos;email&apos;: &apos;william.schultz@mongodb.com&apos;, &apos;username&apos;: &apos;will62794&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-44068&quot; title=&quot;Investigate the slowest sections of RollbackTest and remove any wasted downtime&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-44068&quot;&gt;&lt;del&gt;SERVER-44068&lt;/del&gt;&lt;/a&gt; Add a &apos;restartHeartbeats&apos; replication test command&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/a697d23b924a47084501caee38365970f89852d3&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/a697d23b924a47084501caee38365970f89852d3&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3037816" author="xgen-internal-githook" created="Mon, 13 Apr 2020 21:58:05 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;William Schultz&apos;, &apos;email&apos;: &apos;william.schultz@mongodb.com&apos;, &apos;username&apos;: &apos;will62794&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-44068&quot; title=&quot;Investigate the slowest sections of RollbackTest and remove any wasted downtime&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-44068&quot;&gt;&lt;del&gt;SERVER-44068&lt;/del&gt;&lt;/a&gt; Add RollbackTest control test&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/60102e6a083fe3290e30bbad0c83c6293ba8b301&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/60102e6a083fe3290e30bbad0c83c6293ba8b301&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3037815" author="xgen-internal-githook" created="Mon, 13 Apr 2020 21:58:02 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;William Schultz&apos;, &apos;email&apos;: &apos;william.schultz@mongodb.com&apos;, &apos;username&apos;: &apos;will62794&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-44068&quot; title=&quot;Investigate the slowest sections of RollbackTest and remove any wasted downtime&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-44068&quot;&gt;&lt;del&gt;SERVER-44068&lt;/del&gt;&lt;/a&gt; Add log messages to measure transition durations in RollbackTest&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/f72ac59661db885c5715a33f703076ae1a23f436&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/f72ac59661db885c5715a33f703076ae1a23f436&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3034486" author="william.schultz" created="Fri, 10 Apr 2020 02:54:43 +0000"  >&lt;p&gt;This patch&apos;s goal to is to remove any remaining speed bottlenecks in RollbackTest. I added log messages to measure the durations of each RollbackTest transition method, and added a control test that does a &quot;no-op&quot; rollback sequence so that we can measure the overhead of the fixture accurately. My goal is to measure and optimize the time spent inside the transition methods since that is the only part of a test that the RollbackTest fixture can control. If expensive/slow operations are run in between transitions that will make the test slow but is not the fault of RollbackTest.&lt;/p&gt;

&lt;p&gt;The speed improvements focus on the slowest sections of RollbackTest. In a &lt;a href=&quot;https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_62_64_bit_replica_sets_patch_2f6cac5d4b98969621b48decd2cadaed9e8faf14_5e8f85bee3c3313c88820bbe_20_04_09_20_30_15##%257B%2522compare%2522%253A%255B%257B%2522hash%2522%253A%25222f6cac5d4b98969621b48decd2cadaed9e8faf14%2522%257D%255D%257D&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;baseline run&lt;/a&gt; of rollback_test_control.js, before any improvements, the durations of the transition methods are as follows:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:00.516+0000 [RollbackTest] RollbackTest transition to kRollbackOps took 14 ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:01.349+0000 [RollbackTest] RollbackTest transition to kSyncSourceOpsBeforeRollback took 835 ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:01.350+0000 [RollbackTest] RollbackTest transition to kSyncSourceOpsDuringRollback took 1 ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:04.197+0000 [RollbackTest] RollbackTest transition to kSteadyStateOps took 2847 ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:04.214+0000 [RollbackTest] RollbackTest transition to kStopped took 16 ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;The &apos;kSyncSourceOpsBeforeRollback&apos; transition and the &apos;kSteadyStateOps&apos; transition are the slowest. We can look at each of them separately to understand why they are slow.&lt;/p&gt;

&lt;h3&gt;&lt;a name=&quot;kSyncSourceOpsBeforeRollback&quot;&gt;&lt;/a&gt;kSyncSourceOpsBeforeRollback&lt;/h3&gt;

&lt;p&gt;The &apos;kSyncSourceOpsBeforeRollback&apos; transition is made slow primarily by the election that occurs to switch primary from the rollback node to the sync source node. The sync source node repeatedly fails to run for election even in response to an explicit stepUp request because it cannot see a majority. &lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:00.534+0000 [RollbackTest] Waiting for the new primary ip-10-122-24-2:20271 to be elected&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:00.534+0000 d20281| 2020-04-09T20:41:00.534+00:00 I  COMMAND  21581   [conn2] &quot;Received replSetStepUp request&quot;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:00.534+0000 d20281| 2020-04-09T20:41:00.534+00:00 I  REPL     21485   [conn2] &quot;Canceling catchup takeover callback&quot;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:00.534+0000 d20281| 2020-04-09T20:41:00.534+00:00 I  ELECTION 4615657 [conn2] &quot;Not starting an election for a replSetStepUp request, since we are not electable&quot;,&quot;attr&quot;:{&quot;reason&quot;:&quot;Not standing for election because I cannot see a majority (mask 0x1)&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:00.746+0000 d20281| 2020-04-09T20:41:00.735+00:00 I  COMMAND  21581   [conn2] &quot;Received replSetStepUp request&quot;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:00.746+0000 d20281| 2020-04-09T20:41:00.735+00:00 I  ELECTION 4615657 [conn2] &quot;Not starting an election for a replSetStepUp request, since we are not electable&quot;,&quot;attr&quot;:{&quot;reason&quot;:&quot;Not standing for election because I cannot see a majority (mask 0x1)&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:00.937+0000 d20281| 2020-04-09T20:41:00.936+00:00 I  COMMAND  21581   [conn2] &quot;Received replSetStepUp request&quot;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:00.937+0000 d20281| 2020-04-09T20:41:00.936+00:00 I  ELECTION 4615657 [conn2] &quot;Not starting an election for a replSetStepUp request, since we are not electable&quot;,&quot;attr&quot;:{&quot;reason&quot;:&quot;Not standing for election because I cannot see a majority (mask 0x1)&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;This continues until the sync source node (20281) finally receives a heartbeat from the tiebreaker node (20282) which alerts it of a healthy majority. Then the next stepUp attempt succeeds and the node gets elected:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:01.034+0000 d20281| 2020-04-09T20:41:01.033+00:00 D2 REPL_HB  4615620 [ReplCoord-0] &quot;Received response to heartbeat&quot;,&quot;attr&quot;:{&quot;requestId&quot;:46,&quot;target&quot;:&quot;ip-10-122-24-2:20272&quot;,&quot;response&quot;:{&quot;ok&quot;:1.0,&quot;state&quot;:2,&quot;v&quot;:3,&quot;configTerm&quot;:1,&quot;set&quot;:&quot;RollbackTest&quot;,&quot;syncingTo&quot;:&quot;ip-10-122-24-2:20270&quot;,&quot;term&quot;:1,&quot;primaryId&quot;:0,&quot;durableOpTime&quot;:{&quot;ts&quot;:{&quot;$timestamp&quot;:{&quot;t&quot;:1586464860,&quot;i&quot;:2}},&quot;t&quot;:1},&quot;durableWallTime&quot;:{&quot;$date&quot;:&quot;2020-04-09T20:41:00.178Z&quot;},&quot;opTime&quot;:{&quot;ts&quot;:{&quot;$timestamp&quot;:{&quot;t&quot;:1586464860,&quot;i&quot;:2}},&quot;t&quot;:1},&quot;wallTime&quot;:{&quot;$date&quot;:&quot;2020-04-09T20:41:00.178Z&quot;},&quot;$replData&quot;:{&quot;term&quot;:1,&quot;lastOpCommitted&quot;:{&quot;ts&quot;:{&quot;$timestamp&quot;:{&quot;t&quot;:1586464860,&quot;i&quot;:2}},&quot;t&quot;:1},&quot;lastCommittedWall&quot;:{&quot;$date&quot;:&quot;2020-04-09T20:41:00.178Z&quot;},&quot;lastOpVisible&quot;:{&quot;ts&quot;:{&quot;$timestamp&quot;:{&quot;t&quot;:1586464860,&quot;i&quot;:2}},&quot;t&quot;:1},&quot;configVersion&quot;:3,&quot;configTerm&quot;:1,&quot;replicaSetId&quot;:{&quot;$oid&quot;:&quot;5e8f885b087643361a3fe7fb&quot;},&quot;primaryIndex&quot;:0,&quot;syncSourceIndex&quot;:0,&quot;isPrimary&quot;:false},&quot;$clusterTime&quot;:{&quot;clusterTime&quot;:{&quot;$timestamp&quot;:{&quot;t&quot;:1586464860,&quot;i&quot;:2}},&quot;signature&quot;:{&quot;hash&quot;:{&quot;$binary&quot;:{&quot;base64&quot;:&quot;AAAAAAAAAAAAAAAAAAAAAAAAAAA=&quot;,&quot;subType&quot;:&quot;0&quot;}},&quot;keyId&quot;:0}},&quot;operationTime&quot;:{&quot;$timestamp&quot;:{&quot;t&quot;:1586464860,&quot;i&quot;:2}}}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:01.138+0000 d20281| 2020-04-09T20:41:01.138+00:00 I  COMMAND  21581   [conn2] &quot;Received replSetStepUp request&quot;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:01.138+0000 d20281| 2020-04-09T20:41:01.138+00:00 I  ELECTION 4615661 [conn2] &quot;Starting an election due to step up request&quot;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:01.139+0000 d20281| 2020-04-09T20:41:01.139+00:00 I  ELECTION 21444   [ReplCoord-5] &quot;Dry election run succeeded, running for election&quot;,&quot;attr&quot;:{&quot;newTerm&quot;:2}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:01.156+0000 d20281| 2020-04-09T20:41:01.155+00:00 I  REPL     21331   [OplogApplier-0] &quot;Transition to primary complete; database writes are now permitted&quot;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:01.349+0000 [RollbackTest] Elected the old secondary ip-10-122-24-2:20271 as the new primary&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;The test fixture is not aware of the newly elected primary, however, until another ~200ms after the transition to primary completes, which appears to be due to the default retry interval of the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/04a2c9acc7ca061fb86736b377b897b11f6c7c48/jstests/replsets/libs/rollback_test.js#L277&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;getPrimary&lt;/a&gt; call. To resolve this slowness, we want to immediately trigger a round of heartbeats after &lt;a href=&quot;https://github.com/mongodb/mongo/blob/04a2c9acc7ca061fb86736b377b897b11f6c7c48/jstests/replsets/libs/rollback_test.js#L445&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;reconnecting the sync source node&lt;/a&gt; and also ensure that the test fixture promptly elects a primary and learns of it by reducing the retry interval of &lt;a href=&quot;https://github.com/mongodb/mongo/blob/04a2c9acc7ca061fb86736b377b897b11f6c7c48/jstests/replsets/libs/rollback_test.js#L271-L274&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;these replSetStepUp attempts&lt;/a&gt; and of this &lt;a href=&quot;https://github.com/mongodb/mongo/blob/04a2c9acc7ca061fb86736b377b897b11f6c7c48/jstests/replsets/libs/rollback_test.js#L277&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;getPrimary call&lt;/a&gt;.&lt;/p&gt;

&lt;h3&gt;&lt;a name=&quot;kSteadyStateOps&quot;&gt;&lt;/a&gt;kSteadyStateOps&lt;/h3&gt;

&lt;p&gt;The &apos;kSteadyStateOps&apos; transition is slow for a few different reasons. First, we focus on &lt;a href=&quot;https://github.com/mongodb/mongo/blob/04a2c9acc7ca061fb86736b377b897b11f6c7c48/jstests/replsets/libs/rollback_test.js#L321-L350&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;the block of code&lt;/a&gt; that waits for the rollback to complete. In the baseline run, this section took 1456ms:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:01.350+0000 [RollbackTest] Wait for ip-10-122-24-2:20270 to finish rollback&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:02.806+0000 [jsTest] [RollbackTest] Rollback on ip-10-122-24-2:20270 completed, reconnecting tiebreaker&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;We fail many iterations of the loop that waits for rollback to complete, and we can see that the rollback starts towards the end of this section, so it&apos;s not the rollback itself that is slow. &lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:01.350+0000 [RollbackTest] Reconnecting the secondary ip-10-122-24-2:20270 so it&apos;ll go into rollback&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:01.350+0000 [RollbackTest] Wait for ip-10-122-24-2:20270 to finish rollback&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:01.354+0000 assert failed : Expected RBID to increment past 1 on ip-10-122-24-2:20270&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:01.354+0000 assert.soonNoExcept caught exception, exception: Error: assert failed : Expected RBID to increment past 1 on ip-10-122-24-2:20270&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:01.566+0000 assert failed : Expected RBID to increment past 1 on ip-10-122-24-2:20270&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:01.566+0000 assert.soonNoExcept caught exception, exception: Error: assert failed : Expected RBID to increment past 1 on ip-10-122-24-2:20270&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:01.766+0000 assert failed : Expected RBID to increment past 1 on ip-10-122-24-2:20270&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:01.766+0000 assert.soonNoExcept caught exception, exception: Error: assert failed : Expected RBID to increment past 1 on ip-10-122-24-2:20270&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:01.971+0000 assert failed : Expected RBID to increment past 1 on ip-10-122-24-2:20270&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:01.972+0000 assert.soonNoExcept caught exception, exception: Error: assert failed : Expected RBID to increment past 1 on ip-10-122-24-2:20270&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:02.177+0000 assert failed : Expected RBID to increment past 1 on ip-10-122-24-2:20270&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:02.177+0000 assert.soonNoExcept caught exception, exception: Error: assert failed : Expected RBID to increment past 1 on ip-10-122-24-2:20270&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:02.382+0000 assert failed : Expected RBID to increment past 1 on ip-10-122-24-2:20270&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:02.383+0000 assert.soonNoExcept caught exception, exception: Error: assert failed : Expected RBID to increment past 1 on ip-10-122-24-2:20270&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:02.533+0000 d20280| 2020-04-09T20:41:02.533+00:00 I  REPL     21088   [BackgroundSync] &quot;Changed sync source&quot;,&quot;attr&quot;:{&quot;oldSyncSource&quot;:&quot;empty&quot;,&quot;newSyncSource&quot;:&quot;ip-10-122-24-2:20271&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:02.536+0000 d20280| 2020-04-09T20:41:02.536+00:00 I  ROLLBACK 21593   [BackgroundSync] &quot;Transition to ROLLBACK&quot;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:02.607+0000 d20280| 2020-04-09T20:41:02.607+00:00 I  REPL     21088   [BackgroundSync] &quot;Changed sync source&quot;,&quot;attr&quot;:{&quot;oldSyncSource&quot;:&quot;empty&quot;,&quot;newSyncSource&quot;:&quot;ip-10-122-24-2:20271&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:02.806+0000 [jsTest] [RollbackTest] Rollback on ip-10-122-24-2:20270 completed, reconnecting tiebreaker&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;It is instead taking a long time for the rollback node to find the sync source that it is going to roll back against after we &lt;a href=&quot;https://github.com/mongodb/mongo/blob/04a2c9acc7ca061fb86736b377b897b11f6c7c48/jstests/replsets/libs/rollback_test.js#L491&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;reconnect it&lt;/a&gt;. This is a byproduct of the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/b36c69c5930d25a8f5ae348a2b2fb24f27f925e6/src/mongo/db/repl/bgsync.cpp#L439-L451&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;various 1 second sleeps&lt;/a&gt; in bgsync.cpp that limit how fast we can re-select a sync source. This can be resolved by utilizing the &apos;forceBgSyncSyncSourceRetryWaitMS&apos; failpoint to make these sleeps much shorter. The time it takes for the fixture to detect rollback completion is also limited by the retry interval of the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/04a2c9acc7ca061fb86736b377b897b11f6c7c48/jstests/replsets/libs/rollback_test.js#L322&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;assert.soonNoExcept loop&lt;/a&gt;, so these retry intervals can be shortened as well.&lt;/p&gt;

&lt;p&gt;After rollback has completed, there is another slow section where we reconnect the tiebreaker and wait for it to replicate the latest ops, which takes 627ms from the time we reconnect the node to the time we finish waiting for replication:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:02.806+0000 [jsTest] [RollbackTest] Rollback on ip-10-122-24-2:20270 completed, reconnecting tiebreaker&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:03.247+0000 d20282| 2020-04-09T20:41:03.247+00:00 I  REPL     21088   [BackgroundSync] &quot;Changed sync source&quot;,&quot;attr&quot;:{&quot;oldSyncSource&quot;:&quot;ip-10-122-24-2:20270&quot;,&quot;newSyncSource&quot;:&quot;ip-10-122-24-2:20271&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:03.433+0000 [jsTest] [RollbackTest] awaitReplication completed&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Once again, this is slow because of how long it takes for the tiebreaker node to re-establish a sync source after being re-connected. This appears to be caused by the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/b36c69c5930d25a8f5ae348a2b2fb24f27f925e6/src/mongo/db/repl/bgsync.cpp#L265&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;1 second sleep&lt;/a&gt; that is triggered inside the stopReplProducer failpoint. After we disable the failpoint, it may take up to a second for us to move past it. Furthermore, the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/04a2c9acc7ca061fb86736b377b897b11f6c7c48/jstests/replsets/libs/rollback_test.js#L356&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;awaitReplication&lt;/a&gt; retry interval limits how fast the fixture can detect newly replicated ops, so its retry interval should be shortened.&lt;/p&gt;

&lt;p&gt;After we wait for replication on the tiebreaker, we do data consistency checks, which may take a non negligible amount of time i.e. several hundred milliseconds. We are not focused on speeding up consistency checks themselves in this ticket, so we will leave those as is. &lt;/p&gt;

&lt;p&gt;Finally, we call &lt;a href=&quot;https://github.com/mongodb/mongo/blob/04a2c9acc7ca061fb86736b377b897b11f6c7c48/jstests/replsets/libs/rollback_test.js#L375&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;stopServerReplication&lt;/a&gt; on the tiebreaker node as the final step of the transition to kSteadyStateOps. The time taken between actually re-enabling the failpoint and when we complete the transition is around 303ms. This appears to be a consequence of the 300ms retry interval &lt;a href=&quot;https://github.com/mongodb/mongo/blob/1f1a897967924e2142d011c20600385d0cff5d4b/src/mongo/shell/check_log.js#L125&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;used by checkLog.contains&lt;/a&gt;. That can be reduced to detect the new failpoint more quickly. &lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:03.894+0000 d20282| 2020-04-09T20:41:03.893+00:00 W  CONTROL  23829   [conn2] &quot;Set failpoint&quot;,&quot;attr&quot;:{&quot;failPointName&quot;:&quot;stopReplProducer&quot;,&quot;failPoint&quot;:{&quot;mode&quot;:1,&quot;data&quot;:{},&quot;timesEntered&quot;:4}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:rollback_test_control] 2020-04-09T20:41:04.197+0000 [RollbackTest] RollbackTest transition to kSteadyStateOps took 2847 ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="2487202" author="william.schultz" created="Thu, 17 Oct 2019 17:23:45 +0000"  >&lt;p&gt;For some crude metrics on this, here are filtered logs showing all &quot;transition to ROLLBACK from SECONDARY&quot; and &quot;transition to SECONDARY from ROLLBACK&quot; log messages in an Evergreen run of replica_sets on an &lt;tt&gt;enterprise-rhel-62-64-bit&lt;/tt&gt; variant:  &lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/234077/234077_rollbacks.txt&quot; title=&quot;rollbacks.txt attached to SERVER-44068&quot;&gt;rollbacks.txt&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;. And, to be more accurate, here are only tests that actually use RollbackTest:  &lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/234120/234120_rollbacks_with_rollback_test.txt&quot; title=&quot;rollbacks_with_rollback_test.txt attached to SERVER-44068&quot;&gt;rollbacks_with_rollback_test.txt&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="1088596">SERVER-45627</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                                        </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="1325846">SERVER-47702</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1328489">SERVER-47767</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1330624">SERVER-47814</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="257702" name="after_stats_kSteadyStateOps.png" size="124007" author="william.schultz@mongodb.com" created="Mon, 27 Apr 2020 22:10:05 +0000"/>
                            <attachment id="257704" name="after_stats_kSyncSourceOpsBeforeRollback.png" size="130587" author="william.schultz@mongodb.com" created="Mon, 27 Apr 2020 22:10:05 +0000"/>
                            <attachment id="257701" name="before_stats_kSteadyStateOps.png" size="128409" author="william.schultz@mongodb.com" created="Mon, 27 Apr 2020 22:10:05 +0000"/>
                            <attachment id="257703" name="before_stats_kSyncSourceOpsBeforeRollback.png" size="127230" author="william.schultz@mongodb.com" created="Mon, 27 Apr 2020 22:10:05 +0000"/>
                            <attachment id="234077" name="rollbacks.txt" size="18901" author="william.schultz@mongodb.com" created="Thu, 17 Oct 2019 17:23:41 +0000"/>
                            <attachment id="234120" name="rollbacks_with_rollback_test.txt" size="11877" author="william.schultz@mongodb.com" created="Thu, 17 Oct 2019 18:18:14 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="18953"><![CDATA[v4.4]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 13 Apr 2020 21:58:02 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 41 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-1664</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 41 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>william.schultz@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hvxxfr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hu47zr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3769">Repl 2020-04-20</customfieldvalue>
    <customfieldvalue id="3881">Repl 2020-05-04</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hvxjp3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>