<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:04:03 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-43766] Investigate the slowest sections of ReplSetTest.initiate and remove any wasted downtime</title>
                <link>https://jira.mongodb.org/browse/SERVER-43766</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Investigate the performance of ReplSetTest.initiate to identify and remove any areas of wasted downtime. Wasted downtime would be any time spent not doing useful, required work. For example, waiting for data to propagate over the network or waiting for an arbitrary timeout to fire in the server e.g. a heartbeat interval or election timeout. In contrast, waiting for an initial stable checkpoint would not be wasted downtime.&lt;/p&gt;</description>
                <environment></environment>
        <key id="956578">SERVER-43766</key>
            <summary>Investigate the slowest sections of ReplSetTest.initiate and remove any wasted downtime</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="william.schultz@mongodb.com">William Schultz</assignee>
                                    <reporter username="william.schultz@mongodb.com">William Schultz</reporter>
                        <labels>
                    </labels>
                <created>Wed, 2 Oct 2019 19:05:55 +0000</created>
                <updated>Sun, 29 Oct 2023 22:16:33 +0000</updated>
                            <resolved>Tue, 10 Dec 2019 23:22:35 +0000</resolved>
                                                    <fixVersion>4.3.3</fixVersion>
                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>2</watches>
                                                                                                                <comments>
                            <comment id="2606660" author="william.schultz" created="Fri, 13 Dec 2019 14:41:38 +0000"  >&lt;blockquote&gt;&lt;p&gt;1. Is num_nodes the number per shard, or the total number?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;&apos;num_nodes&apos; is the total number of shard nodes, as reported &lt;a href=&quot;https://github.com/mongodb/mongo/blob/35c4be790c1e03898d9c5443f9a33e36f6f40302/src/mongo/shell/shardingtest.js#L1520&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;. It&apos;s not an entirely accurate metric since it doesn&apos;t include config server size and it only counts the total number of shard nodes, regardless of shard replica set size. For example, for 2 shards of 3 nodes &apos;num_nodes&apos; would appear as 6, as would 3 shards of 2 nodes each.&lt;/p&gt;

&lt;blockquote&gt;&lt;p&gt;2. How many shards is this split over?&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;I think my answer to (1) should address this, if I understand the question correctly. The metrics I currently collect do not include the number of shards in the test. I previously tracked this but switched to just track the total number of nodes. &lt;/p&gt;

&lt;blockquote&gt;&lt;p&gt;3. Why do the legends contain different numbers of nodes?&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;Good question. Something I forgot to point out is that &apos;num_nodes&apos; actually has a different meaning in each chart. In the &quot;before&quot; chart &apos;num_nodes&apos; represents the number of shards in the test, whereas in the &quot;after&quot; chart it represents the total number of shard nodes, as explained in my answer to (1). Sorry, I know this is confusing, but it&apos;s a side effect of the fact that the metrics collection strategy changed a bit over the course of the project. If you ignore the coloring of bars entirely, then the charts can be directly compared. &lt;/p&gt;

&lt;p&gt;In hindsight, it might have been better to just stick with measuring the number of shards instead of total shard nodes, since we now expect that startup and initiation time of multi-node replica sets shouldn&apos;t be much worse that a single node replica set, so we would probably care more about correlating shard count to execution times. &lt;/p&gt;</comment>
                            <comment id="2603461" author="judah.schvimer" created="Fri, 13 Dec 2019 02:06:19 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=william.schultz&quot; class=&quot;user-hover&quot; rel=&quot;william.schultz&quot;&gt;william.schultz&lt;/a&gt;, in the above chart:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Is num_nodes the number per shard, or the total number?&lt;/li&gt;
	&lt;li&gt;How many shards is this split over?&lt;/li&gt;
	&lt;li&gt;Why do the legends contain different numbers of nodes?&lt;/li&gt;
&lt;/ol&gt;
</comment>
                            <comment id="2602782" author="william.schultz" created="Thu, 12 Dec 2019 21:57:21 +0000"  >&lt;p&gt;We can also view the overall performance improvement achieved for ShardingTest startup and initiation performance in the &apos;sharding&apos; suite after these changes were committed. The comparison below show the time it takes to start up and initiate a sharded cluster in ShardingTest before any performance improvements from PM-1360 were committed, and the performance after this ticket was done.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://evergreen.mongodb.com/version/5deec43ae3c331527a946440&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Before&lt;/a&gt;&lt;br/&gt;
 &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;240288_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/240288/240288_5deec43ae3c331527a946440%2Centerprise-rhel-62-64-bit%2Ce79e2a5b15d17fe4b1838c0087dc722e3a6e454c%2CstartupAndInitiate.png&quot; title=&quot;5deec43ae3c331527a946440,enterprise-rhel-62-64-bit,e79e2a5b15d17fe4b1838c0087dc722e3a6e454c,startupAndInitiate.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;240288&quot; file-preview-title=&quot;5deec43ae3c331527a946440,enterprise-rhel-62-64-bit,e79e2a5b15d17fe4b1838c0087dc722e3a6e454c,startupAndInitiate.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/240288/_thumb_240288.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://evergreen.mongodb.com/version/5deea767e3c331527a90dddb&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;After&lt;/a&gt;&lt;br/&gt;
 &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;240289_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/240289/240289_5deea767e3c331527a90dddb%2Centerprise-rhel-62-64-bit%2C4f00d6e3d7e54a2d131b8216d4b4ef9bfd7a7c6b%2CstartupAndInitiate.png&quot; title=&quot;5deea767e3c331527a90dddb,enterprise-rhel-62-64-bit,4f00d6e3d7e54a2d131b8216d4b4ef9bfd7a7c6b,startupAndInitiate.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;240289&quot; file-preview-title=&quot;5deea767e3c331527a90dddb,enterprise-rhel-62-64-bit,4f00d6e3d7e54a2d131b8216d4b4ef9bfd7a7c6b,startupAndInitiate.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/240289/_thumb_240289.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;We see the 95th percentile has dropped from 17328ms to 3966ms, a reduction of 14,642ms.&lt;/p&gt;</comment>
                            <comment id="2597920" author="william.schultz" created="Tue, 10 Dec 2019 23:21:02 +0000"  >&lt;p&gt;The overall performance improvement achieved for ReplSetTest performance in the &apos;replica_sets&apos; suite after the &lt;a href=&quot;https://github.com/mongodb/mongo/commit/440ccdca4de3730c9eac22ce68edb055638cd352&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;final commit&lt;/a&gt; is summarized below.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://evergreen.mongodb.com/version/5dc08364e3c3317a73ff6fa6&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Before&lt;/a&gt;&lt;br/&gt;
 &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;240005_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/240005/240005_5dc08364e3c3317a73ff6fa6%2Centerprise-rhel-62-64-bit%2Cbe045feaf3ba4af8037c5baceda2b15cd2498a24%2CinitiateWithNodeZeroAsPrimary.png&quot; title=&quot;5dc08364e3c3317a73ff6fa6,enterprise-rhel-62-64-bit,be045feaf3ba4af8037c5baceda2b15cd2498a24,initiateWithNodeZeroAsPrimary.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;240005&quot; file-preview-title=&quot;5dc08364e3c3317a73ff6fa6,enterprise-rhel-62-64-bit,be045feaf3ba4af8037c5baceda2b15cd2498a24,initiateWithNodeZeroAsPrimary.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/240005/_thumb_240005.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://evergreen.mongodb.com/version/5deff7e2e3c331637cd5fcf1&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;After&lt;/a&gt;&lt;br/&gt;
 &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;240006_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/240006/240006_5deff7e2e3c331637cd5fcf1%2Centerprise-rhel-62-64-bit%2C4f00d6e3d7e54a2d131b8216d4b4ef9bfd7a7c6b%2CinitiateWithNodeZeroAsPrimary.png&quot; title=&quot;5deff7e2e3c331637cd5fcf1,enterprise-rhel-62-64-bit,4f00d6e3d7e54a2d131b8216d4b4ef9bfd7a7c6b,initiateWithNodeZeroAsPrimary.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;240006&quot; file-preview-title=&quot;5deff7e2e3c331637cd5fcf1,enterprise-rhel-62-64-bit,4f00d6e3d7e54a2d131b8216d4b4ef9bfd7a7c6b,initiateWithNodeZeroAsPrimary.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/240006/_thumb_240006.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;We see a drop in the 95th percentile from 6336ms to 824ms. &lt;/p&gt;</comment>
                            <comment id="2597918" author="xgen-internal-githook" created="Tue, 10 Dec 2019 23:19:47 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;email&apos;: &apos;william.schultz@mongodb.com&apos;, &apos;name&apos;: &apos;William Schultz&apos;, &apos;username&apos;: &apos;will62794&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-43766&quot; title=&quot;Investigate the slowest sections of ReplSetTest.initiate and remove any wasted downtime&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-43766&quot;&gt;&lt;del&gt;SERVER-43766&lt;/del&gt;&lt;/a&gt; Additional optimizations for ReplSetTest initiate speed&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/440ccdca4de3730c9eac22ce68edb055638cd352&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/440ccdca4de3730c9eac22ce68edb055638cd352&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2594642" author="william.schultz" created="Mon, 9 Dec 2019 19:56:14 +0000"  >&lt;p&gt;After adding further optimizations which include reducing this &lt;a href=&quot;https://github.com/mongodb/mongo/blob/ccecd50087d22d90df71ab3c5dd4a58905590307/src/mongo/db/repl/bgsync.cpp#L384&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;bgsync sleep&lt;/a&gt; and waiting for replication inside initiateWithAnyNodeAsPrimary before we turn &lt;a href=&quot;https://github.com/mongodb/mongo/blob/6b013542da50c6f6258f13c135ca03fa1a7c2b38/src/mongo/shell/replsettest.js#L1337&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;up the heartbeat interval again&lt;/a&gt;, here are the initial improvements &lt;a href=&quot;https://github.com/mongodb/mongo/commit/f5a2d477761f5d954ea63a8c8a6cfa02d124e4a7&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;over the original commit&lt;/a&gt;:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://evergreen.mongodb.com/version/5de952fee3c33113884c24cc&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Before&lt;/a&gt;:&lt;br/&gt;
 &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;239828_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/239828/239828_5de952fee3c33113884c24cc%2Centerprise-rhel-62-64-bit%2C3232fb0d0ada81eaee63c02887f6639284da1905%2CinitiateWithNodeZeroAsPrimary.png&quot; title=&quot;5de952fee3c33113884c24cc,enterprise-rhel-62-64-bit,3232fb0d0ada81eaee63c02887f6639284da1905,initiateWithNodeZeroAsPrimary.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;239828&quot; file-preview-title=&quot;5de952fee3c33113884c24cc,enterprise-rhel-62-64-bit,3232fb0d0ada81eaee63c02887f6639284da1905,initiateWithNodeZeroAsPrimary.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/239828/_thumb_239828.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://evergreen.mongodb.com/version/5debc0f9e3c3312ae9e8ff18&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;After&lt;/a&gt;:&lt;br/&gt;
 &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;239829_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/239829/239829_5debc0f9e3c3312ae9e8ff18%2Centerprise-rhel-62-64-bit%2C13944bb3fedc8d91c02c56bb66bb5c76a0a558d0%2CinitiateWithNodeZeroAsPrimary.png&quot; title=&quot;5debc0f9e3c3312ae9e8ff18,enterprise-rhel-62-64-bit,13944bb3fedc8d91c02c56bb66bb5c76a0a558d0,initiateWithNodeZeroAsPrimary.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;239829&quot; file-preview-title=&quot;5debc0f9e3c3312ae9e8ff18,enterprise-rhel-62-64-bit,13944bb3fedc8d91c02c56bb66bb5c76a0a558d0,initiateWithNodeZeroAsPrimary.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/239829/_thumb_239829.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;The 95th percentile shows a significant improvement.&lt;/p&gt;</comment>
                            <comment id="2590905" author="william.schultz" created="Fri, 6 Dec 2019 23:08:02 +0000"  >&lt;p&gt;This is an additional &lt;a href=&quot;https://evergreen.mongodb.com/version/5dea7681e3c3312ae9e13d13&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Evergreen run&lt;/a&gt; with 1 resmoke job and a reduced &lt;a href=&quot;https://github.com/mongodb/mongo/blob/ccecd50087d22d90df71ab3c5dd4a58905590307/src/mongo/db/repl/bgsync.cpp#L384&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;bgsync sleep&lt;/a&gt;:&lt;/p&gt;

&lt;p&gt; &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;239668_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/239668/239668_5dea7681e3c3312ae9e13d13%2Centerprise-rhel-62-64-bit%2C13944bb3fedc8d91c02c56bb66bb5c76a0a558d0.png&quot; title=&quot;5dea7681e3c3312ae9e13d13,enterprise-rhel-62-64-bit,13944bb3fedc8d91c02c56bb66bb5c76a0a558d0.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;239668&quot; file-preview-title=&quot;5dea7681e3c3312ae9e13d13,enterprise-rhel-62-64-bit,13944bb3fedc8d91c02c56bb66bb5c76a0a558d0.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/239668/_thumb_239668.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;And here is another &lt;a href=&quot;https://evergreen.mongodb.com/version/5debc0f9e3c3312ae9e8ff18&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;run&lt;/a&gt; where we reduce the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/ccecd50087d22d90df71ab3c5dd4a58905590307/src/mongo/db/repl/bgsync.cpp#L384&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;bgsync sleep&lt;/a&gt; and also awaitReplication and wait for nodes to agree on their view of the replica set before turning up the heartbeat interval again:&lt;/p&gt;

&lt;p&gt; &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;239786_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/239786/239786_5debc0f9e3c3312ae9e8ff18%2Centerprise-rhel-62-64-bit%2C13944bb3fedc8d91c02c56bb66bb5c76a0a558d0.png&quot; title=&quot;5debc0f9e3c3312ae9e8ff18,enterprise-rhel-62-64-bit,13944bb3fedc8d91c02c56bb66bb5c76a0a558d0.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;239786&quot; file-preview-title=&quot;5debc0f9e3c3312ae9e8ff18,enterprise-rhel-62-64-bit,13944bb3fedc8d91c02c56bb66bb5c76a0a558d0.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/239786/_thumb_239786.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;
</comment>
                            <comment id="2589630" author="william.schultz" created="Fri, 6 Dec 2019 14:54:03 +0000"  >&lt;p&gt;Upon some further inspection, it appears that tests that disable chaining (e.g. &lt;a href=&quot;https://github.com/mongodb/mongo/blob/ccecd50087d22d90df71ab3c5dd4a58905590307/jstests/replsets/step_down_during_draining2.js#L25&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;step_down_during_draining2.js&lt;/a&gt;) may be the culprit for the slower bunch of 3 node tests that lie a bit above the 1 second duration point. If we reduce &lt;a href=&quot;https://github.com/mongodb/mongo/blob/ccecd50087d22d90df71ab3c5dd4a58905590307/src/mongo/db/repl/bgsync.cpp#L384&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this sleep&lt;/a&gt; in bgsync.cpp and run a &lt;a href=&quot;https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_62_64_bit_replica_sets_patch_f5a2d477761f5d954ea63a8c8a6cfa02d124e4a7_5de9d266e3c3311388563bce_19_12_06_04_00_39##%257B%2522compare%2522%253A%255B%257B%2522hash%2522%253A%2522f5a2d477761f5d954ea63a8c8a6cfa02d124e4a7%2522%257D%255D%257D&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;new patch&lt;/a&gt;, we get the following profile:&lt;/p&gt;

&lt;p&gt; &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;239590_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/239590/239590_5de9d266e3c3311388563bce%2Centerprise-rhel-62-64-bit%2Cf5a2d477761f5d954ea63a8c8a6cfa02d124e4a7.png&quot; title=&quot;5de9d266e3c3311388563bce,enterprise-rhel-62-64-bit,f5a2d477761f5d954ea63a8c8a6cfa02d124e4a7.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;239590&quot; file-preview-title=&quot;5de9d266e3c3311388563bce,enterprise-rhel-62-64-bit,f5a2d477761f5d954ea63a8c8a6cfa02d124e4a7.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/239590/_thumb_239590.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;This appears to bring those 3 node tests into the lower, sub 1 second cluster of tests. There is still a small cluster of 2 and 3 node tests that lie above the 2.5 second mark, and it would be worth doing further investigation to see if that is just noise or there is some other issue making all of those tests slower.&lt;/p&gt;</comment>
                            <comment id="2588121" author="william.schultz" created="Fri, 6 Dec 2019 03:36:42 +0000"  >&lt;p&gt;As &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=judah.schvimer&quot; class=&quot;user-hover&quot; rel=&quot;judah.schvimer&quot;&gt;judah.schvimer&lt;/a&gt; pointed out, the standard deviation even after the changes is still somewhat high (2281ms). This may be skewed somewhat by the outliers in the data. If we take the mean and standard deviation of the lowest 95%, 85%, and 75% of the data, we see the following stats, which give a bit more accurate picture of the variance in the data:&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Lowest 95%&lt;/em&gt;&lt;br/&gt;
 mean: 832ms&lt;br/&gt;
 stddev: 704ms&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Lowest 85%&lt;/em&gt;&lt;br/&gt;
 mean: 638ms&lt;br/&gt;
 stddev: 413ms&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Lowest 75%&lt;/em&gt;&lt;br/&gt;
 mean: 514ms&lt;br/&gt;
 stddev: 246ms&lt;/p&gt;

&lt;p&gt;As discussed offline, there is still an apparent correlation between initiate time and replica set size. My thought is that this may have to do with the slowness of the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/d78c2f80751777ddcd4ebd1148c0c7a482b53e63/src/mongo/shell/replsettest.js#L1358&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;stepUp&lt;/a&gt; function inside initiateWithNodeZeroAsPrimary. I may consider trying to speed this up in a follow up patch.&#160;&lt;/p&gt;

&lt;p&gt;Also, here is a histogram from a &lt;a href=&quot;https://evergreen.mongodb.com/version/5de98193e3c33113884ffb22&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;sequential Evergreen run&lt;/a&gt; i.e. only using 1 resmoke job. This data appears to have less outlier noise:&lt;br/&gt;
 &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;239558_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/239558/239558_5de98193e3c33113884ffb22%2Centerprise-rhel-62-64-bit%2Cf5a2d477761f5d954ea63a8c8a6cfa02d124e4a7.png&quot; title=&quot;5de98193e3c33113884ffb22,enterprise-rhel-62-64-bit,f5a2d477761f5d954ea63a8c8a6cfa02d124e4a7.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;239558&quot; file-preview-title=&quot;5de98193e3c33113884ffb22,enterprise-rhel-62-64-bit,f5a2d477761f5d954ea63a8c8a6cfa02d124e4a7.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/239558/_thumb_239558.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2586942" author="william.schultz" created="Thu, 5 Dec 2019 21:51:35 +0000"  >&lt;p&gt;Here are initial measurements of the performance impact of these changes. Here are histograms showing &lt;tt&gt;initiateWithNodeZeroAsPrimary&lt;/tt&gt; durations &lt;a href=&quot;https://evergreen.mongodb.com/version/5dc08364e3c3317a73ff6fa6&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;before the change&lt;/a&gt;:&lt;br/&gt;
 &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;239517_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/239517/239517_5dc08364e3c3317a73ff6fa6%2Centerprise-rhel-62-64-bit%2Cbe045feaf3ba4af8037c5baceda2b15cd2498a24.png&quot; title=&quot;5dc08364e3c3317a73ff6fa6,enterprise-rhel-62-64-bit,be045feaf3ba4af8037c5baceda2b15cd2498a24.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;239517&quot; file-preview-title=&quot;5dc08364e3c3317a73ff6fa6,enterprise-rhel-62-64-bit,be045feaf3ba4af8037c5baceda2b15cd2498a24.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/239517/_thumb_239517.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;and these are the associated summary stats for &lt;tt&gt;initiateWithNodeZeroAsPrimary&lt;/tt&gt; duration:&lt;/p&gt;

&lt;p&gt;median=3477ms&lt;br/&gt;
mean=3955ms&lt;br/&gt;
stddev=2338ms&lt;br/&gt;
95th percentile: 6335ms&lt;br/&gt;
75th percentile: 5281ms&lt;br/&gt;
50th percentile: 3477ms&lt;/p&gt;

&lt;p&gt;Here is the duration histogram &lt;a href=&quot;https://evergreen.mongodb.com/version/5de952fee3c33113884c24cc&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;after the changes&lt;/a&gt;:&lt;/p&gt;

&lt;p&gt; &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;239518_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/239518/239518_5de952fee3c33113884c24cc%2Centerprise-rhel-62-64-bit%2C3232fb0d0ada81eaee63c02887f6639284da1905.png&quot; title=&quot;5de952fee3c33113884c24cc,enterprise-rhel-62-64-bit,3232fb0d0ada81eaee63c02887f6639284da1905.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;239518&quot; file-preview-title=&quot;5de952fee3c33113884c24cc,enterprise-rhel-62-64-bit,3232fb0d0ada81eaee63c02887f6639284da1905.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/239518/_thumb_239518.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt; &lt;br/&gt;
along with the summary stats:&lt;/p&gt;

&lt;p&gt;median=641ms&lt;br/&gt;
mean=1196ms&lt;br/&gt;
stddev=2281ms&lt;br/&gt;
95th percentile: 3685ms&lt;br/&gt;
75th percentile: 1288ms&lt;br/&gt;
50th percentile: 641ms&lt;/p&gt;

&lt;p&gt;To more accurately measure the improvements here, I would still like to run a full sequential test run, to measure single threaded test execution times.&lt;/p&gt;</comment>
                            <comment id="2586893" author="xgen-internal-githook" created="Thu, 5 Dec 2019 21:27:05 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;William Schultz&apos;, &apos;username&apos;: &apos;will62794&apos;, &apos;email&apos;: &apos;william.schultz@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-43766&quot; title=&quot;Investigate the slowest sections of ReplSetTest.initiate and remove any wasted downtime&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-43766&quot;&gt;&lt;del&gt;SERVER-43766&lt;/del&gt;&lt;/a&gt; Speed up initial syncs and awaitLastStableRecoveryTimestamp in ReplSetTest initiate&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/f5a2d477761f5d954ea63a8c8a6cfa02d124e4a7&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/f5a2d477761f5d954ea63a8c8a6cfa02d124e4a7&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2579716" author="xgen-internal-githook" created="Tue, 3 Dec 2019 15:01:13 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;email&apos;: &apos;william.schultz@mongodb.com&apos;, &apos;name&apos;: &apos;William Schultz&apos;, &apos;username&apos;: &apos;will62794&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-43766&quot; title=&quot;Investigate the slowest sections of ReplSetTest.initiate and remove any wasted downtime&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-43766&quot;&gt;&lt;del&gt;SERVER-43766&lt;/del&gt;&lt;/a&gt; Prevent use of failpoints in ReplSetTest when using random binary versions for nodes&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/a0831c75a97b935f30c2987a67a3ee6c92fab49c&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/a0831c75a97b935f30c2987a67a3ee6c92fab49c&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2575780" author="xgen-internal-githook" created="Mon, 2 Dec 2019 23:40:45 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;William Schultz&apos;, &apos;username&apos;: &apos;will62794&apos;, &apos;email&apos;: &apos;william.schultz@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-43766&quot; title=&quot;Investigate the slowest sections of ReplSetTest.initiate and remove any wasted downtime&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-43766&quot;&gt;&lt;del&gt;SERVER-43766&lt;/del&gt;&lt;/a&gt; Speed up primary drain mode in ReplSetTest initiate&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/2e948c4e94b17089ab56a5437447f9988c31103d&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/2e948c4e94b17089ab56a5437447f9988c31103d&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2560533" author="william.schultz" created="Thu, 21 Nov 2019 22:16:12 +0000"  >&lt;h3&gt;&lt;a name=&quot;initiateawaitLastStableRecoveryTimestamp%281149msfor5nodes%29&quot;&gt;&lt;/a&gt;initiate awaitLastStableRecoveryTimestamp (1149ms for 5 nodes)&lt;/h3&gt;

&lt;p&gt;If we look at what one of the nodes is doing between the completion of awaitSecondaryNodes and the completion of awaitLastStableRecoveryTimestamp, it again appears that sync source selection is slowing things down. We start waiting for a stable recovery timestamp at 25:32.607, and node d21523, a secondary, does not pick a sync source until 25:33.535, almost a full second later. Shortly after we finish waiting for the recovery timestamp and we see that the &apos;appendOplogNote&apos; completed with a duration of 920ms:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T17:25:32.579+0000 ReplSetTest initiate awaitSecondaryNodes took 1237ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T17:25:32.607+0000 AwaitLastStableRecoveryTimestamp: Beginning for [&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T17:25:32.634+0000 AwaitLastStableRecoveryTimestamp: ensuring the commit point advances for [&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T17:25:33.535+0000 d21523| 2019-11-20T17:25:33.534+0000 I  REPL     [BackgroundSync] sync source candidate: ip-10-122-75-91:21520&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T17:25:33.536+0000 d21523| 2019-11-20T17:25:33.535+0000 I  REPL     [BackgroundSync] Changed sync source from empty to ip-10-122-75-91:21520&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T17:25:33.539+0000 d21520| 2019-11-20T17:25:33.539+0000 I  COMMAND  [conn1] command admin.$cmd appName: &quot;MongoDB Shell&quot; command: appendOplogNote { appendOplogNote: 1.0, data: { awaitLastStableRecoveryTimestamp: 1.0 }, writeConcern: { w: &quot;majority&quot;, wtimeout: 600000.0 }, lsid: { id: UUID(&quot;5a39fc0b-d164-49ca-9fe6-f62eda611178&quot;) }, $clusterTime: { clusterTime: Timestamp(1574270731, 6), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $readPreference: { mode: &quot;secondaryPreferred&quot; }, $db: &quot;admin&quot; } numYields:0 reslen:163 locks:{ ParallelBatchWriterMode: { acquireCount: { r: 1 } }, ReplicationStateTransition: { acquireCount: { w: 1 } }, Global: { acquireCount: { w: 1 } } } flowControl:{ acquireCount: 1 } storage:{} protocol:op_msg 920ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T17:25:33.547+0000 AwaitLastStableRecoveryTimestamp: Waiting for stable recovery timestamps for [&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T17:25:33.547+0000 AwaitLastStableRecoveryTimestamp: ip-10-122-75-91:21521 does not have a stable recovery timestamp yet.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T17:25:33.751+0000 AwaitLastStableRecoveryTimestamp: A stable recovery timestamp has successfully established on [&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T17:25:33.751+0000 ReplSetTest initiate awaitLastStableRecoveryTimestamp took 1149ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;This seems to implicate &lt;a href=&quot;https://github.com/mongodb/mongo/blob/98c73ff5aa5a151483a2e69b15dd60172238ceba/src/mongo/db/repl/bgsync.cpp#L397&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;one of the 1 second sleeps&lt;/a&gt; scattered through bgsync.cpp. If we reduce all of these sleeps to 100 milliseconds, we can reduce the duration of this section somewhat, as seen in this &lt;a href=&quot;https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_62_64_bit_replica_sets_patch_60c957304e503dbca360838627cf0f8402764929_5dd57e5ae3c3315db829d31e_19_11_20_17_56_43#/%23%257B%2522compare%2522%253A%255B%257B%2522hash%2522%253A%252260c957304e503dbca360838627cf0f8402764929%2522%257D%255D%257D#%257B%2522compare%2522%253A%255B%257B%2522hash%2522%253A%252260c957304e503dbca360838627cf0f8402764929%2522%257D%255D%257D&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;patch&lt;/a&gt;:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T18:21:45.071+0000 ReplSetTest initiate awaitSecondaryNodes took 1236ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T18:21:45.534+0000 d21771| 2019-11-20T18:21:45.533+0000 I  REPL     [BackgroundSync] sync source candidate: ip-10-122-75-130:21770&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T18:21:45.535+0000 d21771| 2019-11-20T18:21:45.534+0000 I  REPL     [BackgroundSync] Changed sync source from empty to ip-10-122-75-130:21770&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T18:21:45.540+0000 d21773| 2019-11-20T18:21:45.539+0000 I  REPL     [BackgroundSync] sync source candidate: ip-10-122-75-130:21771&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T18:21:45.541+0000 d21773| 2019-11-20T18:21:45.541+0000 I  REPL     [BackgroundSync] Changed sync source from empty to ip-10-122-75-130:21771&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T18:21:45.547+0000 d21774| 2019-11-20T18:21:45.546+0000 I  REPL     [BackgroundSync] sync source candidate: ip-10-122-75-130:21773&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T18:21:45.548+0000 d21774| 2019-11-20T18:21:45.548+0000 I  REPL     [BackgroundSync] Changed sync source from empty to ip-10-122-75-130:21773&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T18:21:45.642+0000 d21772| 2019-11-20T18:21:45.642+0000 I  REPL     [BackgroundSync] sync source candidate: ip-10-122-75-130:21771&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T18:21:45.644+0000 d21772| 2019-11-20T18:21:45.643+0000 I  REPL     [BackgroundSync] Changed sync source from empty to ip-10-122-75-130:21771&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T18:21:45.758+0000 ReplSetTest initiate awaitLastStableRecoveryTimestamp took 667ms for 5 nodes&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;This a 482ms over the baseline. Initial sync source selection still seems to take several 100 milliseconds, though, even with the reduce bgsync.cpp sleeps i.e. consider the timestamp different between the &quot;ReplSetTest initiate awaitSecondaryNodes took 1236ms&quot; message and the first &quot;sync source candidate: ip-10-122-75-130:21770&quot; message from node d21771. This seemed to suggest that slow heartbeats may prevent the nodes from receiving the information they need to pick a sync source sooner. If we run a &lt;a href=&quot;https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_62_64_bit_replica_sets_patch_60c957304e503dbca360838627cf0f8402764929_5dd59dcce3c3315db82c9aef_19_11_20_20_10_54#/%23%257B%2522compare%2522%253A%255B%257B%2522hash%2522%253A%252260c957304e503dbca360838627cf0f8402764929%2522%257D%255D%257D#%257B%2522compare%2522%253A%255B%257B%2522hash%2522%253A%252260c957304e503dbca360838627cf0f8402764929%2522%257D%255D%257D&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;new patch build&lt;/a&gt; with both reduced bgsync sleeps and a heartbeat interval of 500ms, we see greater improvements:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T20:48:32.911+0000 ReplSetTest initiate awaitSecondaryNodes took 1442ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T20:48:32.938+0000 AwaitLastStableRecoveryTimestamp: Beginning for [&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T20:48:32.950+0000 AwaitLastStableRecoveryTimestamp: ensuring the commit point advances for [&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T20:48:33.087+0000 d20523| 2019-11-20T20:48:33.087+0000 I  REPL     [BackgroundSync] sync source candidate: ip-10-122-70-126:20520&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T20:48:33.089+0000 d20521| 2019-11-20T20:48:33.089+0000 I  REPL     [BackgroundSync] sync source candidate: ip-10-122-70-126:20520&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T20:48:33.090+0000 d20524| 2019-11-20T20:48:33.090+0000 I  REPL     [BackgroundSync] sync source candidate: ip-10-122-70-126:20520&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T20:48:33.093+0000 d20520| 2019-11-20T20:48:33.093+0000 I  COMMAND  [conn1] command admin.$cmd appName: &quot;MongoDB Shell&quot; command: appendOplogNote { appendOplogNote: 1.0, data: { awaitLastStableRecoveryTimestamp: 1.0 }, writeConcern: { w: &quot;majority&quot;, wtimeout: 600000.0 }, lsid: { id: UUID(&quot;bfbac1a0-5327-4542-9f2c-9d56b7ae7104&quot;) }, $clusterTime: { clusterTime: Timestamp(1574282911, 6), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $readPreference: { mode: &quot;secondaryPreferred&quot; }, $db: &quot;admin&quot; } numYields:0 reslen:163 locks:{ ParallelBatchWriterMode: { acquireCount: { r: 1 } }, ReplicationStateTransition: { acquireCount: { w: 1 } }, Global: { acquireCount: { w: 1 } } } flowControl:{ acquireCount: 1 } storage:{} protocol:op_msg 142ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T20:48:33.093+0000 AwaitLastStableRecoveryTimestamp: Waiting for stable recovery timestamps for [&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T20:48:33.094+0000 d20522| 2019-11-20T20:48:33.094+0000 I  REPL     [BackgroundSync] sync source candidate: ip-10-122-70-126:20520&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T20:48:33.104+0000 AwaitLastStableRecoveryTimestamp: A stable recovery timestamp has successfully established on [&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T20:48:33.104+0000 ReplSetTest initiate awaitLastStableRecoveryTimestamp took 168ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;This reduces the execution time to 168ms, a 982ms improvement over the baseline.&lt;/p&gt;



</comment>
                            <comment id="2560485" author="william.schultz" created="Thu, 21 Nov 2019 21:52:46 +0000"  >&lt;h3&gt;&lt;a name=&quot;initiateawaitSecondaryNodes%281237msfor5nodes%29&quot;&gt;&lt;/a&gt;initiate awaitSecondaryNodes (1237ms for 5 nodes)&lt;/h3&gt;

&lt;p&gt;If we look at what nodes are doing between the completion of the &apos;initiate command&apos; and the completion of &apos;awaitSecondaryNodes&apos;, we can see that initial sync of each node becomes a candidate for what is slowing things down:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T17:25:31.333+0000 ReplSetTest initiate command took 1066ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T17:25:31.398+0000 d21522| 2019-11-20T17:25:31.397+0000 D1 INITSYNC [ReplCoordExtern-0] Error getting sync source: &apos;InvalidSyncSource: No valid sync source available. Our last fetched optime: { ts: Timestamp(0, 0), t: -1 }&apos;, trying again in 1000ms at 2019-11-20T17:25:32.397+0000. Attempt 1 of 60&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T17:25:31.403+0000 d21521| 2019-11-20T17:25:31.403+0000 D1 INITSYNC [ReplCoordExtern-0] Error getting sync source: &apos;InvalidSyncSource: No valid sync source available. Our last fetched optime: { ts: Timestamp(0, 0), t: -1 }&apos;, trying again in 1000ms at 2019-11-20T17:25:32.403+0000. Attempt 1 of 60&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T17:25:31.403+0000 d21523| 2019-11-20T17:25:31.403+0000 D1 INITSYNC [ReplCoordExtern-0] Error getting sync source: &apos;InvalidSyncSource: No valid sync source available. Our last fetched optime: { ts: Timestamp(0, 0), t: -1 }&apos;, trying again in 1000ms at 2019-11-20T17:25:32.403+0000. Attempt 1 of 60&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T17:25:31.403+0000 d21524| 2019-11-20T17:25:31.403+0000 D1 INITSYNC [ReplCoordExtern-0] Error getting sync source: &apos;InvalidSyncSource: No valid sync source available. Our last fetched optime: { ts: Timestamp(0, 0), t: -1 }&apos;, trying again in 1000ms at 2019-11-20T17:25:32.403+0000. Attempt 1 of 60&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T17:25:32.397+0000 d21522| 2019-11-20T17:25:32.397+0000 I  REPL     [ReplCoordExtern-1] sync source candidate: ip-10-122-75-91:21520&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T17:25:32.403+0000 d21524| 2019-11-20T17:25:32.403+0000 I  REPL     [ReplCoordExtern-1] sync source candidate: ip-10-122-75-91:21520&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T17:25:32.403+0000 d21523| 2019-11-20T17:25:32.403+0000 I  REPL     [ReplCoordExtern-1] sync source candidate: ip-10-122-75-91:21520&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T17:25:32.403+0000 d21521| 2019-11-20T17:25:32.403+0000 I  REPL     [ReplCoordExtern-1] sync source candidate: ip-10-122-75-91:21520&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T17:25:32.521+0000 d21522| 2019-11-20T17:25:32.521+0000 I  INITSYNC [ReplCoordExtern-0] initial sync done; took 1s.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T17:25:32.532+0000 d21524| 2019-11-20T17:25:32.532+0000 I  INITSYNC [ReplCoordExtern-0] initial sync done; took 1s.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T17:25:32.532+0000 d21521| 2019-11-20T17:25:32.532+0000 I  INITSYNC [ReplCoordExtern-2] initial sync done; took 1s.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T17:25:32.533+0000 d21523| 2019-11-20T17:25:32.533+0000 I  INITSYNC [ReplCoordExtern-2] initial sync done; took 1s.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T17:25:32.579+0000 ReplSetTest initiate awaitSecondaryNodes took 1237ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;After we see the &quot;Error getting sync source&quot; message from each node, each node takes 1 second until it logs a &quot;sync source candidate&quot; message. This seems to implicate this &lt;a href=&quot;https://github.com/mongodb/mongo/blob/98c73ff5aa5a151483a2e69b15dd60172238ceba/src/mongo/db/repl/initial_syncer.h#L92-L94&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;hard-coded initial sync timeout&lt;/a&gt;, which is set at 1 second. If we run another &lt;a href=&quot;https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_62_64_bit_replica_sets_patch_60c957304e503dbca360838627cf0f8402764929_5dd57c41e3c3315db829b3c7_19_11_20_17_47_46#/%23%257B%2522compare%2522%253A%255B%257B%2522hash%2522%253A%252260c957304e503dbca360838627cf0f8402764929%2522%257D%255D%257D#%257B%2522compare%2522%253A%255B%257B%2522hash%2522%253A%252260c957304e503dbca360838627cf0f8402764929%2522%257D%255D%257D&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;patch build&lt;/a&gt; with this interval reduced to 50 milliseconds, we slightly improve the duration here, but not drastically:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T18:17:24.344+0000 ReplSetTest initiate command took 1083ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T18:17:24.408+0000 d22021| 2019-11-20T18:17:24.408+0000 D1 INITSYNC [ReplCoordExtern-0] Error getting sync source: &apos;InvalidSyncSource: No valid sync source available. Our last fetched optime: { ts: Timestamp(0, 0), t: -1 }&apos;, trying again in 50ms at 2019-11-20T18:17:24.458+0000. Attempt 1 of 60&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T18:17:24.458+0000 d22021| 2019-11-20T18:17:24.458+0000 D1 INITSYNC [ReplCoordExtern-1] Error getting sync source: &apos;InvalidSyncSource: No valid sync source available. Our last fetched optime: { ts: Timestamp(0, 0), t: -1 }&apos;, trying again in 50ms at 2019-11-20T18:17:24.508+0000. Attempt 2 of 60&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T18:17:24.508+0000 d22021| 2019-11-20T18:17:24.508+0000 D1 INITSYNC [ReplCoordExtern-0] Error getting sync source: &apos;InvalidSyncSource: No valid sync source available. Our last fetched optime: { ts: Timestamp(0, 0), t: -1 }&apos;, trying again in 50ms at 2019-11-20T18:17:24.558+0000. Attempt 3 of 60&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T18:17:24.558+0000 d22021| 2019-11-20T18:17:24.558+0000 D1 INITSYNC [ReplCoordExtern-1] Error getting sync source: &apos;InvalidSyncSource: No valid sync source available. Our last fetched optime: { ts: Timestamp(0, 0), t: -1 }&apos;, trying again in 50ms at 2019-11-20T18:17:24.608+0000. Attempt 4 of 60&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T18:17:24.608+0000 d22021| 2019-11-20T18:17:24.608+0000 D1 INITSYNC [ReplCoordExtern-0] Error getting sync source: &apos;InvalidSyncSource: No valid sync source available. Our last fetched optime: { ts: Timestamp(0, 0), t: -1 }&apos;, trying again in 50ms at 2019-11-20T18:17:24.658+0000. Attempt 5 of 60&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T18:17:24.658+0000 d22021| 2019-11-20T18:17:24.658+0000 D1 INITSYNC [ReplCoordExtern-1] Error getting sync source: &apos;InvalidSyncSource: No valid sync source available. Our last fetched optime: { ts: Timestamp(0, 0), t: -1 }&apos;, trying again in 50ms at 2019-11-20T18:17:24.708+0000. Attempt 6 of 60&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T18:17:24.708+0000 d22021| 2019-11-20T18:17:24.708+0000 D1 INITSYNC [ReplCoordExtern-0] Error getting sync source: &apos;InvalidSyncSource: No valid sync source available. Our last fetched optime: { ts: Timestamp(0, 0), t: -1 }&apos;, trying again in 50ms at 2019-11-20T18:17:24.758+0000. Attempt 7 of 60&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T18:17:24.758+0000 d22021| 2019-11-20T18:17:24.758+0000 D1 INITSYNC [ReplCoordExtern-1] Error getting sync source: &apos;InvalidSyncSource: No valid sync source available. Our last fetched optime: { ts: Timestamp(0, 0), t: -1 }&apos;, trying again in 50ms at 2019-11-20T18:17:24.808+0000. Attempt 8 of 60&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T18:17:24.808+0000 d22021| 2019-11-20T18:17:24.808+0000 D1 INITSYNC [ReplCoordExtern-0] Error getting sync source: &apos;InvalidSyncSource: No valid sync source available. Our last fetched optime: { ts: Timestamp(0, 0), t: -1 }&apos;, trying again in 50ms at 2019-11-20T18:17:24.858+0000. Attempt 9 of 60&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T18:17:24.858+0000 d22021| 2019-11-20T18:17:24.858+0000 D1 INITSYNC [ReplCoordExtern-1] Error getting sync source: &apos;InvalidSyncSource: No valid sync source available. Our last fetched optime: { ts: Timestamp(0, 0), t: -1 }&apos;, trying again in 50ms at 2019-11-20T18:17:24.908+0000. Attempt 10 of 60&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T18:17:24.908+0000 d22021| 2019-11-20T18:17:24.908+0000 I  REPL     [ReplCoordExtern-0] sync source candidate: ip-10-122-76-160:22020&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T18:17:25.079+0000 d22021| 2019-11-20T18:17:25.078+0000 I  INITSYNC [ReplCoordExtern-0] initial sync done; took 0s.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T18:17:25.182+0000 ReplSetTest initiate awaitSecondaryNodes took 830ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;We see that even with a reduced retry interval, a node (d22021) is still stuck waiting to find a sync source for a while (~500ms). This implies that the node may not be able to pick a valid sync source because it has not updated its view of other nodes in the replica set yet. If we reduce the heartbeat interval to 500ms in addition to reducing the initial sync retry intervals to 50ms in another &lt;a href=&quot;https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_62_64_bit_replica_sets_patch_60c957304e503dbca360838627cf0f8402764929_5dd59cdae3c3315db82c751e_19_11_20_20_06_51#/%23%257B%2522compare%2522%253A%255B%257B%2522hash%2522%253A%252260c957304e503dbca360838627cf0f8402764929%2522%257D%255D%257D#%257B%2522compare%2522%253A%255B%257B%2522hash%2522%253A%252260c957304e503dbca360838627cf0f8402764929%2522%257D%255D%257D&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;patch&lt;/a&gt;, we see that a node can find a sync source more quickly, and the overall time of this section is reduced further:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T20:47:42.220+0000 ReplSetTest initiate command took 1080ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T20:47:42.292+0000 d22271| 2019-11-20T20:47:42.291+0000 D1 INITSYNC [ReplCoordExtern-0] Error getting sync source: &apos;InvalidSyncSource: No valid sync source available. Our last fetched optime: { ts: Timestamp(0, 0), t: -1 }&apos;, trying again in 50ms at 2019-11-20T20:47:42.341+0000. Attempt 1 of 60&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T20:47:42.341+0000 d22271| 2019-11-20T20:47:42.341+0000 D1 INITSYNC [ReplCoordExtern-1] Error getting sync source: &apos;InvalidSyncSource: No valid sync source available. Our last fetched optime: { ts: Timestamp(0, 0), t: -1 }&apos;, trying again in 50ms at 2019-11-20T20:47:42.391+0000. Attempt 2 of 60&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T20:47:42.391+0000 d22271| 2019-11-20T20:47:42.391+0000 I  REPL     [ReplCoordExtern-0] sync source candidate: ip-10-122-73-89:22270&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T20:47:42.540+0000 d22271| 2019-11-20T20:47:42.540+0000 I  INITSYNC [ReplCoordExtern-0] initial sync done; took 0s.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T20:47:42.647+0000 ReplSetTest initiate awaitSecondaryNodes took 418ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;The execution time here of 418ms is a 819ms reduction over the baseline. Initial sync itself may be the limiting factor in reducing this duration further, but it might be worth a bit more investigation. We should also run these tests a few times to make sure we&apos;re not measuring noise.&lt;/p&gt;


</comment>
                            <comment id="2560451" author="william.schultz" created="Thu, 21 Nov 2019 21:37:40 +0000"  >&lt;h3&gt;&lt;a name=&quot;initiatecommand%281066msfor5nodes%29&quot;&gt;&lt;/a&gt;initiate command (1066ms for 5 nodes)&lt;/h3&gt;

&lt;p&gt;In this case it&apos;s pretty easy to locate the offending log messages:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T17:25:30.316+0000 d21520| 2019-11-20T17:25:30.316+0000 I  REPL     [ReplCoord-1] Stopping replication producer&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T17:25:31.313+0000 d21520| 2019-11-20T17:25:31.313+0000 I  REPL     [ReplBatcher] Oplog buffer has been drained in term 1&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;This seems a clear indicator that &lt;a href=&quot;https://github.com/mongodb/mongo/blob/6c9e9014b8829c040c73d8313444e5013569b91c/src/mongo/db/repl/opqueue_batcher.cpp#L298&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this 1 second sleep&lt;/a&gt; is slowing the initiate command down. We can test this theory by running a &lt;a href=&quot;https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_62_64_bit_replica_sets_patch_60c957304e503dbca360838627cf0f8402764929_5dd57ac2e3c3315db8299ddd_19_11_20_17_41_23#/%23%257B%2522compare%2522%253A%255B%257B%2522hash%2522%253A%252260c957304e503dbca360838627cf0f8402764929%2522%257D%255D%257D#%257B%2522compare%2522%253A%255B%257B%2522hash%2522%253A%252260c957304e503dbca360838627cf0f8402764929%2522%257D%255D%257D&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;new patch build&lt;/a&gt; with that sleep reduced from 1 second to 50 milliseconds. In that patch, we see the initiate command duration significantly reduced:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T18:12:36.601+0000 ReplSetTest initiate command took 279ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;and it seems clear that the primary drain speed had a large impact on this:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T18:12:36.398+0000 d22770| 2019-11-20T18:12:36.396+0000 I  REPL     [ReplCoord-0] Stopping replication producer&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-20T18:12:36.398+0000 d22770| 2019-11-20T18:12:36.396+0000 I  REPL     [ReplBatcher] Oplog buffer has been drained in term 1&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;This is a speed improvement of 787ms over the baseline. If we run another &lt;a href=&quot;https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_62_64_bit_replica_sets_patch_60c957304e503dbca360838627cf0f8402764929_5ddb5068e3c3317c8491485a_19_11_25_03_54_17##%257B%2522compare%2522%253A%255B%257B%2522hash%2522%253A%252260c957304e503dbca360838627cf0f8402764929%2522%257D%255D%257D&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;patch build&lt;/a&gt; with additional logging, we can see what is contributing to the remaining execution time:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-25T04:45:58.425+0000 ReplSetTest initiate getPrimary took 210ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-25T04:45:58.425+0000 ReplSetTest initiate command took 279ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;This &lt;a href=&quot;https://github.com/mongodb/mongo/blob/9bb0231ccb28efe34a4bcd430049b85cc1b32ea0/src/mongo/shell/replsettest.js#L1063&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;getPrimary call&lt;/a&gt; takes over 200ms, presumably because it fails on its first try, and then &lt;a href=&quot;https://github.com/mongodb/mongo/blob/2e5391cb4ca41bb352507915b0fe24b142d61a29/src/mongo/shell/assert.js#L327&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;waits 200ms&lt;/a&gt; until it tries again. If we reduce this getPrimary retry interval to 25ms (&lt;a href=&quot;https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_62_64_bit_replica_sets_patch_60c957304e503dbca360838627cf0f8402764929_5ddb57b9e3c3317c8491594c_19_11_25_04_25_30##%257B%2522compare%2522%253A%255B%257B%2522hash%2522%253A%252260c957304e503dbca360838627cf0f8402764929%2522%257D%255D%257D&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;patch build&lt;/a&gt;), we bring the initiate command time down to 103ms, a 963ms improvement over the baseline:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-25T05:10:12.793+0000 ReplSetTest initiate getPrimary took 34ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:replsettest_control_1_node] 2019-11-25T05:10:12.794+0000 ReplSetTest initiate command took 103ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
</comment>
                            <comment id="2560446" author="william.schultz" created="Thu, 21 Nov 2019 21:35:13 +0000"  >&lt;p&gt;To investigate the slowest sections of ReplSetTest.initiate and how they could be improved, I started from the baseline metrics shown in the above comment. The execution of ReplSetTest.initiate can be divided into the time spent in &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a417e979908af2124b990d68a22c437005877790/src/mongo/shell/replsettest.js#L1216&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;initiateWithAnyNodeAsPrimary&lt;/a&gt; and the time spent inside the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a417e979908af2124b990d68a22c437005877790/src/mongo/shell/replsettest.js#L1218-L1222&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;stepUp&lt;/a&gt; call inside initiateWithNodeZeroAsPrimary. Here are the slowest sections of initiateWithAnyNodeAsPrimary, extracted from the Evergreen baseline above:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2019-11-20T17:25:31.333+0000 ReplSetTest initiate command took 1066ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2019-11-20T17:25:32.579+0000 ReplSetTest initiate awaitSecondaryNodes took 1237ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2019-11-20T17:25:33.751+0000 ReplSetTest initiate awaitLastStableRecoveryTimestamp took 1149ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;and these are the slowest sections of stepUp&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2019-11-20T17:25:35.642+0000 ReplSetTest stepUp awaitNodesAgreeOnAppliedOpTime took 1865ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;We look at each below to determine why they are slow. &lt;/p&gt;








</comment>
                            <comment id="2554084" author="william.schultz" created="Wed, 20 Nov 2019 17:31:08 +0000"  >&lt;p&gt;Here are baseline performance metrics of ReplSetTest.initiate from an Evergreen RHEL 6.2 &lt;a href=&quot;https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_62_64_bit_replica_sets_patch_60c957304e503dbca360838627cf0f8402764929_5dd57092e3c3315db828caaa_19_11_20_16_57_55##%257B%2522compare%2522%253A%255B%257B%2522hash%2522%253A%252260c957304e503dbca360838627cf0f8402764929%2522%257D%255D%257D&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;patch build&lt;/a&gt;:&lt;/p&gt;


&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2019-11-20T17:25:30.261+0000 ReplSetTest startSet took 826ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2019-11-20T17:25:31.333+0000 ReplSetTest initiate command took 1066ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2019-11-20T17:25:31.344+0000 ReplSetTest reconfiguration to add nodes took 9ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2019-11-20T17:25:32.579+0000 ReplSetTest initiate awaitSecondaryNodes took 1237ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2019-11-20T17:25:32.607+0000 ReplSetTest initiate waitForKeys took 5ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2019-11-20T17:25:33.751+0000 ReplSetTest initiate awaitLastStableRecoveryTimestamp took 1149ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2019-11-20T17:25:33.751+0000 ReplSetTest initiateWithAnyNodeAsPrimary took 3490ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2019-11-20T17:25:33.776+0000 ReplSetTest stepUp awaitReplication took 25ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2019-11-20T17:25:35.642+0000 ReplSetTest stepUp awaitNodesAgreeOnAppliedOpTime took 1865ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2019-11-20T17:25:35.648+0000 ReplSetTest stepUp awaitNodesAgreeOnPrimary took 7ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2019-11-20T17:25:35.650+0000 ReplSetTest initiate stepUp took 1899ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2019-11-20T17:25:35.650+0000 ReplSetTest initiateWithNodeZeroAsPrimary took 5390ms for 5 nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="2549469" author="william.schultz" created="Tue, 19 Nov 2019 17:22:03 +0000"  >&lt;p&gt;Some areas of the code to investigate that may slow down the ReplSetTest.initiate process:&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/98c73ff5aa5a151483a2e69b15dd60172238ceba/src/mongo/db/repl/bgsync.cpp#L233&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;1 second sleeps in bgsync.cpp&lt;/a&gt;&lt;/li&gt;
	&lt;li&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/98c73ff5aa5a151483a2e69b15dd60172238ceba/src/mongo/db/repl/initial_syncer.h#L92-L99&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Retry timeouts for initial sync&lt;/a&gt;&lt;/li&gt;
	&lt;li&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/0dc87ca907a725308d21378dcf65380a64f31364/src/mongo/db/repl/repl_set_config.cpp#L56&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Default heartbeat timeout of 2 seconds&lt;/a&gt;&lt;/li&gt;
	&lt;li&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/6c9e9014b8829c040c73d8313444e5013569b91c/src/mongo/db/repl/opqueue_batcher.cpp#L298&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;1 second sleep in op queue batcher&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10520">
                    <name>Problem/Incident</name>
                                            <outwardlinks description="causes">
                                        <issuelink>
            <issuekey id="1113921">SERVER-45765</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="1052602">SERVER-45151</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1324723">SERVER-47691</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="956937">SERVER-43774</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="240005" name="5dc08364e3c3317a73ff6fa6,enterprise-rhel-62-64-bit,be045feaf3ba4af8037c5baceda2b15cd2498a24,initiateWithNodeZeroAsPrimary.png" size="139828" author="william.schultz@mongodb.com" created="Tue, 10 Dec 2019 23:11:26 +0000"/>
                            <attachment id="239517" name="5dc08364e3c3317a73ff6fa6,enterprise-rhel-62-64-bit,be045feaf3ba4af8037c5baceda2b15cd2498a24.png" size="148878" author="william.schultz@mongodb.com" created="Thu, 5 Dec 2019 21:43:05 +0000"/>
                            <attachment id="239828" name="5de952fee3c33113884c24cc,enterprise-rhel-62-64-bit,3232fb0d0ada81eaee63c02887f6639284da1905,initiateWithNodeZeroAsPrimary.png" size="142687" author="william.schultz@mongodb.com" created="Mon, 9 Dec 2019 19:55:45 +0000"/>
                            <attachment id="239518" name="5de952fee3c33113884c24cc,enterprise-rhel-62-64-bit,3232fb0d0ada81eaee63c02887f6639284da1905.png" size="153827" author="william.schultz@mongodb.com" created="Thu, 5 Dec 2019 21:44:28 +0000"/>
                            <attachment id="239558" name="5de98193e3c33113884ffb22,enterprise-rhel-62-64-bit,f5a2d477761f5d954ea63a8c8a6cfa02d124e4a7.png" size="148114" author="william.schultz@mongodb.com" created="Fri, 6 Dec 2019 04:22:35 +0000"/>
                            <attachment id="239590" name="5de9d266e3c3311388563bce,enterprise-rhel-62-64-bit,f5a2d477761f5d954ea63a8c8a6cfa02d124e4a7.png" size="150190" author="william.schultz@mongodb.com" created="Fri, 6 Dec 2019 14:52:04 +0000"/>
                            <attachment id="239668" name="5dea7681e3c3312ae9e13d13,enterprise-rhel-62-64-bit,13944bb3fedc8d91c02c56bb66bb5c76a0a558d0.png" size="147198" author="william.schultz@mongodb.com" created="Fri, 6 Dec 2019 23:07:56 +0000"/>
                            <attachment id="239829" name="5debc0f9e3c3312ae9e8ff18,enterprise-rhel-62-64-bit,13944bb3fedc8d91c02c56bb66bb5c76a0a558d0,initiateWithNodeZeroAsPrimary.png" size="139138" author="william.schultz@mongodb.com" created="Mon, 9 Dec 2019 19:56:10 +0000"/>
                            <attachment id="239786" name="5debc0f9e3c3312ae9e8ff18,enterprise-rhel-62-64-bit,13944bb3fedc8d91c02c56bb66bb5c76a0a558d0.png" size="149721" author="william.schultz@mongodb.com" created="Mon, 9 Dec 2019 16:14:41 +0000"/>
                            <attachment id="240289" name="5deea767e3c331527a90dddb,enterprise-rhel-62-64-bit,4f00d6e3d7e54a2d131b8216d4b4ef9bfd7a7c6b,startupAndInitiate.png" size="144735" author="william.schultz@mongodb.com" created="Thu, 12 Dec 2019 21:57:18 +0000"/>
                            <attachment id="240288" name="5deec43ae3c331527a946440,enterprise-rhel-62-64-bit,e79e2a5b15d17fe4b1838c0087dc722e3a6e454c,startupAndInitiate.png" size="139032" author="william.schultz@mongodb.com" created="Thu, 12 Dec 2019 21:57:08 +0000"/>
                            <attachment id="240287" name="5deec43ae3c331527a946440,enterprise-rhel-62-64-bit,e79e2a5b15d17fe4b1838c0087dc722e3a6e454c,startupAndInitiate.png" size="139032" author="william.schultz@mongodb.com" created="Thu, 12 Dec 2019 21:56:55 +0000"/>
                            <attachment id="240006" name="5deff7e2e3c331637cd5fcf1,enterprise-rhel-62-64-bit,4f00d6e3d7e54a2d131b8216d4b4ef9bfd7a7c6b,initiateWithNodeZeroAsPrimary.png" size="139913" author="william.schultz@mongodb.com" created="Tue, 10 Dec 2019 23:12:02 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>19.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 2 Dec 2019 23:40:45 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 8 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-1360</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 8 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>48.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>judah.schvimer@mongodb.com</customfieldvalue>
            <customfieldvalue>william.schultz@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hvva2v:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hu47zz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3436">Repl 2019-12-02</customfieldvalue>
    <customfieldvalue id="3437">Repl 2019-12-16</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hvuwc7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>