<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:54:06 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-40130] Improve multi-threading</title>
                <link>https://jira.mongodb.org/browse/SERVER-40130</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hello, at sendinblue we use mongodb since a long time and have a few clusters running with big datasets.&lt;/p&gt;

&lt;p&gt;We currently struggle with our cluster using Wiretiger, slow start, replication not able to succeed or is very long....&#160;&lt;/p&gt;

&lt;p&gt;Here is information about the sizing of cluster we have here is some stats :&#160;&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;mongodb version 3.4.17&lt;/li&gt;
	&lt;li&gt;10 shards&lt;/li&gt;
	&lt;li&gt;databases per shard : ~ 26K&lt;/li&gt;
	&lt;li&gt;collections per shard : ~ 600K&lt;/li&gt;
	&lt;li&gt;shard size : ~ 500G&lt;/li&gt;
	&lt;li&gt;files in data directory for the tested shard :&#160;&#160;1 873 414&#160; (find /data/ | wc -l)
	&lt;ul&gt;
		&lt;li&gt;FYI we didn&apos;t split journal, data, indexes&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;I have currently isolated one shard of a cluster to do some debugging about the bottlenek we encounter. I troubleshoot issues on a secondary which is currently on a google cloud&#160; instance seems to freeze a lot while starting, play oplogs ...&lt;/p&gt;



&lt;p&gt;We identified that the shard looks like to have some process running as mono thread or not efficently multi-threaded. This instance run in instance of 16 vCPU at 2.5GHz and 96G of memory.&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;At the starting of the mongod instance it take very long time and&#160; statistics on server seems to show that one or 2 vCPU are effectively working.&#160;&lt;/p&gt;

&lt;p&gt;We have found some relative information here : &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-27700?focusedCommentId=1480933&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-1480933&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-27700?focusedCommentId=1480933&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-1480933&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;But as we already use a version that have the improvments, we shouldn&apos;t strugle on replication op are evictions.&#160;&lt;/p&gt;

&lt;p&gt;Here is the current configuration we use :&#160;&lt;br/&gt;
```STORAGE &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; wiredtiger_open config: create,cache_size=40960M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress)```&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;We have tried some modification of WireTiger based on this document bellow and&#160; the&#160; comment on the previous Jira link : &lt;a href=&quot;https://source.wiredtiger.com/2.9.0/group__wt.html#gab435a7372679c74261cb62624d953300&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://source.wiredtiger.com/2.9.0/group__wt.html#gab435a7372679c74261cb62624d953300&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Currently my configuration is :&#160;&lt;/p&gt;

&lt;p&gt;```net:&lt;br/&gt;
 bindIp: 0.0.0.0&lt;br/&gt;
 port: XXXXX&lt;br/&gt;
processManagement:&lt;br/&gt;
 pidFilePath: /var/run/mongodb/shard1.pid&lt;br/&gt;
replication:&lt;br/&gt;
&#160; oplogSizeMB: 10240&lt;br/&gt;
&#160; &#160;replSetName: XXXXX&lt;br/&gt;
setParameter:&lt;br/&gt;
&#160; cursorTimeoutMillis: 1800000&lt;br/&gt;
&#160; failIndexKeyTooLong: true&lt;br/&gt;
sharding:&lt;br/&gt;
&#160; clusterRole: shardsvr&lt;br/&gt;
storage:&lt;br/&gt;
&#160; dbPath: /data/&lt;br/&gt;
&#160; &#160;engine: wiredTiger&lt;br/&gt;
&#160; &#160;wiredTiger:&lt;br/&gt;
&#160; &#160;engineConfig:&lt;br/&gt;
&#160; &#160; &#160;cacheSizeGB: 40&lt;br/&gt;
```&lt;/p&gt;

&lt;p&gt;Is there any setup that permit to increase the multi processing at startup and on the replication process because seems that some process are not.&#160;&lt;/p&gt;

&lt;p&gt;```shard1-:# ps -T -p 32107&lt;br/&gt;
 PID SPID TTY TIME CMD&lt;br/&gt;
32107 32107 ? 00:05:27 shard&lt;br/&gt;
32107 32109 ? 00:00:00 signalP.gThread&lt;br/&gt;
32107 32110 ? 00:00:00 Backgro.kSource&lt;br/&gt;
32107 32216 ? 00:00:00 shard&lt;br/&gt;
32107 32217 ? 00:00:00&#160;shard&lt;br/&gt;
32107 32218 ? 00:00:00&#160;shard&lt;br/&gt;
32107 32219 ? 00:00:24 shard&lt;br/&gt;
32107 32220 ? 00:00:24 shard&lt;br/&gt;
32107 32221 ? 00:00:24 shard&lt;br/&gt;
32107 32222 ? 00:00:24 shard&lt;br/&gt;
32107 32223 ? 00:00:02 shard&lt;br/&gt;
32107 32224 ? 00:01:03 shard&lt;br/&gt;
32107 32225 ? 00:00:01 WTJourn.Flusher&lt;/p&gt;

&lt;p&gt;```&#160;&lt;/p&gt;

&lt;p&gt;During startup we clearly see that the server is stuck on&#160; this with one process with 100% cpu and the rest doing nothing almost :&lt;/p&gt;

&lt;p&gt;```2019-03-14T17:38:24.611+0000 I STORAGE &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; wiredtiger_open config: create,cache_size=40960M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),file_manager=(close_handle_minimum=10000,close_idle_time=3600,close_scan_interval=10)&lt;br/&gt;
2019-03-14T17:38:42.046+0000 I STORAGE &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; WiredTiger message &lt;span class=&quot;error&quot;&gt;&amp;#91;1552585122:46632&amp;#93;&lt;/span&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;32107:0x7fd0dd295d40&amp;#93;&lt;/span&gt;, txn-recover: Main recovery loop: starting at 401225/10635776&lt;br/&gt;
2019-03-14T17:38:42.047+0000 I STORAGE &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; WiredTiger message &lt;span class=&quot;error&quot;&gt;&amp;#91;1552585122:47626&amp;#93;&lt;/span&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;32107:0x7fd0dd295d40&amp;#93;&lt;/span&gt;, txn-recover: Recovering log 401225 through 401226&lt;br/&gt;
2019-03-14T17:38:42.588+0000 I STORAGE &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; WiredTiger message &lt;span class=&quot;error&quot;&gt;&amp;#91;1552585122:588204&amp;#93;&lt;/span&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;32107:0x7fd0dd295d40&amp;#93;&lt;/span&gt;, &lt;a href=&quot;file:collection-75452--270635807257442042.wt&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;file:collection-75452--270635807257442042.wt&lt;/a&gt;, txn-recover: Recovering log 401226 through 401226&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;ls &lt;del&gt;alih /data/collection-75452&lt;/del&gt;-270635807257442042.wt&lt;br/&gt;
2274678025 &lt;del&gt;rw-r&lt;/del&gt;&lt;del&gt;r&lt;/del&gt;- 1 mongodb mongodb 1.2M Mar 14 17:38 /data/collection-75452-270635807257442042.wt&lt;/p&gt;

&lt;p&gt;free -mh&lt;br/&gt;
 total used free shared buffers cached&lt;br/&gt;
Mem: 94G 61G 32G 32M 85M 41G&lt;br/&gt;
-/+ buffers/cache: 20G 74G&lt;br/&gt;
Swap: 24G 0B 24G&lt;/p&gt;

&lt;p&gt;``` This step take 40min and using only 1 cpu.&#160;&lt;/p&gt;

&lt;p&gt;Can you help on this ? I know you will need more info that i can probably provide.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Thanks in advance&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;


&lt;p&gt;&#160;&lt;/p&gt;</description>
                <environment></environment>
        <key id="716951">SERVER-40130</key>
            <summary>Improve multi-threading</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="eric.sedor@mongodb.com">Eric Sedor</assignee>
                                    <reporter username="kpichardie">Pichardie k&#233;vin</reporter>
                        <labels>
                    </labels>
                <created>Thu, 14 Mar 2019 17:52:52 +0000</created>
                <updated>Thu, 16 May 2019 08:18:45 +0000</updated>
                            <resolved>Wed, 20 Mar 2019 17:54:54 +0000</resolved>
                                    <version>3.4.17</version>
                                                    <component>Performance</component>
                    <component>Storage</component>
                                        <votes>0</votes>
                                    <watches>9</watches>
                                                                                                                <comments>
                            <comment id="2187101" author="kpichardie" created="Thu, 21 Mar 2019 09:11:34 +0000"  >&lt;p&gt;Hello Eric,&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Yes but this case isn&apos;t updated since 2016 :s. Should i update on this case ?&lt;/p&gt;

&lt;p&gt;Can you also share your finding to keep traces of bottleneck, and that we can summup on case&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-25025&quot; title=&quot;Improve startup time when there are tens of thousands of collections/indexes on WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-25025&quot;&gt;&lt;del&gt;SERVER-25025&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Best regards,&lt;/p&gt;

&lt;p&gt;K&#233;vin&lt;/p&gt;</comment>
                            <comment id="2186451" author="eric.sedor" created="Wed, 20 Mar 2019 17:54:54 +0000"  >&lt;p&gt;At this time I&apos;m going to close this ticket as a duplicate. Can you please see &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-25025&quot; title=&quot;Improve startup time when there are tens of thousands of collections/indexes on WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-25025&quot;&gt;&lt;del&gt;SERVER-25025&lt;/del&gt;&lt;/a&gt; for future updates?&lt;/p&gt;</comment>
                            <comment id="2186449" author="eric.sedor" created="Wed, 20 Mar 2019 17:53:34 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kpichardie&quot; class=&quot;user-hover&quot; rel=&quot;kpichardie&quot;&gt;kpichardie&lt;/a&gt;, with these logs we&apos;ve been able to rule out some other possibilities and currently believe that this is a case of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-25025&quot; title=&quot;Improve startup time when there are tens of thousands of collections/indexes on WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-25025&quot;&gt;&lt;del&gt;SERVER-25025&lt;/del&gt;&lt;/a&gt;. However, it&apos;s likely exacerbated by there being 1.8 million files, versus tens of thousands.&lt;/p&gt;</comment>
                            <comment id="2186429" author="eric.sedor" created="Wed, 20 Mar 2019 17:39:48 +0000"  >&lt;p&gt;Thanks K&#233;vin, this helps; We are taking a look and will let you know.&lt;/p&gt;</comment>
                            <comment id="2185965" author="kpichardie" created="Wed, 20 Mar 2019 11:34:16 +0000"  >&lt;p&gt;Hello Eric,&lt;/p&gt;

&lt;p&gt;I have retried with normal stop and time is mostly the same. See log attached on secure protal.&#160;&lt;/p&gt;

&lt;p&gt;Let me know if you need more info.&#160;&lt;/p&gt;

&lt;p&gt;K&#233;vin&lt;/p&gt;</comment>
                            <comment id="2185882" author="kpichardie" created="Wed, 20 Mar 2019 08:15:51 +0000"  >&lt;p&gt;Hello Eric,&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;I have normally stopped the process using systemd but i think it&apos;s also take a long time to stop arround 5 min and i will check but systemd probably killed it.&#160;&lt;/p&gt;

&lt;p&gt;As default timeout is 5 minutes i do think mongo is killed.&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;I will try increase the timeout to check the time to restart.&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;K&#233;vin&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2185331" author="eric.sedor" created="Tue, 19 Mar 2019 20:37:32 +0000"  >&lt;p&gt;Hi K&#233;vin, the bulk of the time spent during restart appears to involve the work necessary to recover from an unclean shutdown. Can you clarify how you are stopping the node?&lt;/p&gt;</comment>
                            <comment id="2185125" author="kpichardie" created="Tue, 19 Mar 2019 17:32:13 +0000"  >&lt;p&gt;Hello Eric,&lt;/p&gt;

&lt;p&gt;Please find the file of the logs during restart.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Best regards,&lt;/p&gt;

&lt;p&gt;K&#233;vin&lt;/p&gt;</comment>
                            <comment id="2185102" author="eric.sedor" created="Tue, 19 Mar 2019 17:19:43 +0000"  >&lt;p&gt;Of course K&#233;vin; I&apos;ve generated an uploader for you &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/8600b7ec-c3d5-4d99-a82f-cfd64345e3ca.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2184612" author="kpichardie" created="Tue, 19 Mar 2019 09:36:05 +0000"  >&lt;p&gt;Hello Eric,&lt;/p&gt;

&lt;p&gt;Can i have a link on secure portal to upload log information.&lt;/p&gt;

&lt;p&gt;Sorry i missed it in comment but it switch as secondary just after STARTUP2&lt;/p&gt;

&lt;p&gt;2019-03-14T23:23:40.186+0000 I REPL &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSync&amp;#93;&lt;/span&gt; transition to SECONDARY&lt;/p&gt;

&lt;p&gt;I will try to do the restart with iostat but we don&apos;t see any limitation on this part.&lt;/p&gt;

&lt;p&gt;I have uploaded the iostat for my restart test, can you provide a secure link for the rest of files ?&lt;/p&gt;

&lt;p&gt;Best regards,&lt;br/&gt;
 K&#233;vin&lt;/p&gt;</comment>
                            <comment id="2184493" author="eric.sedor" created="Mon, 18 Mar 2019 23:22:24 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kpichardie&quot; class=&quot;user-hover&quot; rel=&quot;kpichardie&quot;&gt;kpichardie&lt;/a&gt;, we are tracking some known inefficiencies with high file counts in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-25025&quot; title=&quot;Improve startup time when there are tens of thousands of collections/indexes on WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-25025&quot;&gt;&lt;del&gt;SERVER-25025&lt;/del&gt;&lt;/a&gt; but would like to investigate more thoroughly to see if any other issues are in play. It looks like you may have snipped some log lines from the above comment, is that right? If so, can you please provide the complete logs between 2019-03-14T17:38:24 and a time when the node transitions to SECONDARY status?&lt;/p&gt;

&lt;p&gt;It may also be helpful if you can repeat the collection of diagnostic data during the secondary&apos;s restart while also running the following:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;delay=1&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;iostat -k -t -x ${delay:?} &amp;gt;iostat.log &amp;amp;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;This will allow us to get disk metrics for additional periods during the startup which are not available in the current diagnostic data.&lt;/p&gt;

&lt;p&gt;Thanks in advance!&lt;/p&gt;</comment>
                            <comment id="2183503" author="kpichardie" created="Mon, 18 Mar 2019 09:23:35 +0000"  >&lt;p&gt;Hello,&#160;&lt;/p&gt;

&lt;p&gt;I have uploaded diagnostics from the tests i made last thursday so the 14th and week-end.&lt;/p&gt;

&lt;p&gt;Basically the problem is impacting replication (we frequently have lags) and startup that is slow.&#160;&lt;/p&gt;

&lt;p&gt;Problem is pretty constant on the replication we see several jump in the delay and we see that one of the CPU is pretty high while others are not overloaded.&lt;/p&gt;

&lt;p&gt;But during 14th i was trying some custom wiretiger configuration like explained previously but no success.&lt;/p&gt;

&lt;p&gt;Last restart with classic configuration params took a very long time :&lt;/p&gt;

&lt;p&gt;2019-03-14T17:38:24.599+0000 I CONTROL &lt;span class=&quot;error&quot;&gt;&amp;#91;main&amp;#93;&lt;/span&gt; ***** SERVER RESTARTED *****&lt;br/&gt;
 2019-03-14T17:38:24.607+0000 I CONTROL &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; MongoDB starting : pid=32107 port=XXXX dbpath=/data/ 64-bit host=shard1-2&lt;br/&gt;
 2019-03-14T17:38:24.607+0000 I CONTROL &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; db version v3.4.17&lt;br/&gt;
 2019-03-14T17:38:24.607+0000 I CONTROL &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; git version: 7c14a47868643bb691a507a92fe25541f998eca4&lt;br/&gt;
 2019-03-14T17:38:24.607+0000 I CONTROL &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; OpenSSL version: OpenSSL 1.0.1t 3 May 2016&lt;br/&gt;
 .....&lt;br/&gt;
 2019-03-14T23:23:16.471+0000 W NETWORK &lt;span class=&quot;error&quot;&gt;&amp;#91;ReplicaSetMonitor-TaskExecutor-0&amp;#93;&lt;/span&gt; Failed to connect to XXXXXX:XXXXX, in(checking socket for error after poll), reason: Connection refused&lt;br/&gt;
 2019-03-14T23:23:40.139+0000 I NETWORK &lt;span class=&quot;error&quot;&gt;&amp;#91;thread2&amp;#93;&lt;/span&gt; waiting for connections on port XXXX&lt;br/&gt;
 2019-03-14T23:23:40.183+0000 I REPL &lt;span class=&quot;error&quot;&gt;&amp;#91;replExecDBWorker-0&amp;#93;&lt;/span&gt; New replica set config in use: { _id: &quot;shard1&quot;, version: 88, protocolVersion: 1, members: [ { _id: 2, host: &quot;shard1-arb-1:XXX&quot;, arbiterOnly: true, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 3, host: &quot;shard1-arb-2:XXXX&quot;, arbiterOnly: true, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 8, host: &quot;shard1-1:XXXXX&quot;, arbiterOnly: false, buildIndexes: true, hidden: false, priority: 100.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 9, host: &quot;shard1-1:XXXXX&quot;, arbiterOnly: false, buildIndexes: true, hidden: false, priority: 50.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 10, host: &quot;shard1-2:XXXXX&quot;, arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: 2000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId(&apos;591440486e078e6c1d57d210&apos;) } }&lt;br/&gt;
 2019-03-14T23:23:40.183+0000 I REPL &lt;span class=&quot;error&quot;&gt;&amp;#91;replExecDBWorker-0&amp;#93;&lt;/span&gt; This node is shard1-2:XXXX in the config&lt;br/&gt;
 2019-03-14T23:23:40.183+0000 I REPL &lt;span class=&quot;error&quot;&gt;&amp;#91;replExecDBWorker-0&amp;#93;&lt;/span&gt; transition to STARTUP2&lt;br/&gt;
 2019-03-14T23:23:40.183+0000 I REPL &lt;span class=&quot;error&quot;&gt;&amp;#91;replExecDBWorker-0&amp;#93;&lt;/span&gt; Starting replication storage threads&lt;br/&gt;
 2019-03-14T23:23:40.184+0000 I REPL &lt;span class=&quot;error&quot;&gt;&amp;#91;replExecDBWorker-0&amp;#93;&lt;/span&gt; Starting replication fetcher thread&lt;br/&gt;
 2019-03-14T23:23:40.184+0000 I REPL &lt;span class=&quot;error&quot;&gt;&amp;#91;replExecDBWorker-0&amp;#93;&lt;/span&gt; Starting replication applier thread&lt;br/&gt;
 2019-03-14T23:23:40.184+0000 I REPL &lt;span class=&quot;error&quot;&gt;&amp;#91;replExecDBWorker-0&amp;#93;&lt;/span&gt; Starting replication reporter thread&lt;br/&gt;
 2019-03-14T23:23:40.184+0000 I ASIO &lt;span class=&quot;error&quot;&gt;&amp;#91;NetworkInterfaceASIO-Replication-0&amp;#93;&lt;/span&gt; Connecting to shard1-arb-1:XXXXX&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;I believe that some processes are mono threaded and causing slow processing during startup and for replication as 1 CPU is saturated.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2181188" author="eric.sedor" created="Thu, 14 Mar 2019 20:43:19 +0000"  >&lt;p&gt;Hello &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kpichardie&quot; class=&quot;user-hover&quot; rel=&quot;kpichardie&quot;&gt;kpichardie&lt;/a&gt; and thanks for the information so far. To best help us understand the behavior you&apos;re observing can you please archive (tar or zip) the &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt; directory (described &lt;a href=&quot;https://docs.mongodb.com/manual/administration/analyzing-mongodb-performance/#full-time-diagnostic-data-capture&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;) for all nodes during a representative incident and attach it to this ticket?&lt;/p&gt;

&lt;p&gt;Timestamps for said incidents will help us target our examination of this data.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="300619">SERVER-25025</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="211733" name="iostat.log.gz" size="195732" author="kpichardie" created="Tue, 19 Mar 2019 11:38:09 +0000"/>
                            <attachment id="211612" name="metrics.tar.gz" size="64021473" author="kpichardie" created="Mon, 18 Mar 2019 09:01:01 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>13.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 14 Mar 2019 20:43:19 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 46 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>dmitry.agranat@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 46 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>eric.sedor@mongodb.com</customfieldvalue>
            <customfieldvalue>kpichardie</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|huqu2f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|huggpz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[eric.sedor@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|huqgbr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>