<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:24:28 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-30628] Timeouts between &apos;mongos&apos; and replicaset nodes rejecting requests</title>
                <link>https://jira.mongodb.org/browse/SERVER-30628</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We had several issues in the past few days under our applications, and after further analysis, we found out that they are all related to our running instances of sharded MongoDB. During this examination, we noticed that our servers were reporting a lot of connection timeouts. After analyzing our network we discovered that it didn&apos;t have any performance issues. The tests results are provided along with this ticket in &apos;reportNetwork_mongo-20170811.txt&apos; file.&lt;/p&gt;

&lt;p&gt;Also, we found some links on the web of people having similar problems to this one that we&apos;re facing; this is a group of links that we found:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-24711&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-24711&lt;/a&gt;&lt;br/&gt;
&lt;a href=&quot;https://stackoverflow.com/questions/38485285/mongodb-replica-heartbeat-request-time-exceeded&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://stackoverflow.com/questions/38485285/mongodb-replica-heartbeat-request-time-exceeded&lt;/a&gt;&lt;br/&gt;
&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-24058&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-24058&lt;/a&gt;&lt;br/&gt;
&lt;a href=&quot;https://github.com/rofl0r/proxychains-ng/issues/171&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/rofl0r/proxychains-ng/issues/171&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;But even so, we think we are facing an undocumented problem here (maybe related to those others).&lt;/p&gt;

&lt;p&gt;A description of the problem:&lt;br/&gt;
Some requests made to our sharded clusters give us a timeout message (found at the log from the DB) for no apparent reason or pattern that we could figure out. If we try to perform the same request right after the problematic one, it may works without any problems but sometimes It don&apos;t. We even implemented this retry at our application code to mitigate the problem, but since it takes much time to timeout and make the request again, our client experience some slowness due to this issue and making this retry isn&apos;t a satisfactory fix to the problem for us.&lt;/p&gt;

&lt;p&gt;Currently, we have three different clusters running sharded MongoDB. One of them is operating at Softlayer&#8217;s(IBM) data center, and two others at OVH. We&apos;re going to focus our data provided in this ticket on the one from IBMs despite also having problems on the other two, just because we think that we&apos;re already providing a lot of data to you guys, but if needed we can provide data from the other two clusters.&lt;/p&gt;

&lt;p&gt;The cluster is composed of (all logs will be relative to this cluster):&lt;br/&gt;
Servers hosted in IBM&#8217;s data center.&lt;br/&gt;
3 MongoDB(3.4.4) Servers running in a VM with 32GB RAM and 8 cores.&lt;br/&gt;
3 MongoDB(3.4.4) Configs running in a VM with 2GB RAM and 2 cores.&lt;br/&gt;
8 nodes(3.4.4) with the replication factor of 3 Bare metal with 256GB RAM and 40 cores.&lt;br/&gt;
All of them communicate between each other using 1Gbps network.&lt;/p&gt;

&lt;p&gt;We can say these issues aren&apos;t related to infrastructure problems at the DC, as the all of our clusters are having this issues, regardless of the infrastructure provider. Also, it is important to report that the cluster size is not a factor in this problem because we have a tiny cluster of only two servers and they are facing the same problems.&lt;/p&gt;

&lt;p&gt;Our DBA also tried to fine tune some configurations and had no luck in fixing the problem. The tuned configurations were:&lt;br/&gt;
WiredTigerConcurrentReadOperations &#8211; from default to 256&lt;br/&gt;
WiredTigerConcurrentWriteOperations &#8211; from default to 256&lt;br/&gt;
ShardingTaskExecutorPoolHostTimeoutMS &#8211; from default to 7200000&lt;br/&gt;
ShardingTaskExecutorPoolMinSize &#8211; from default to 10&lt;br/&gt;
taskExecutorPoolSize &#8211; from default to 16 (twice the number of cores)&lt;/p&gt;

&lt;p&gt;We also tried to change the max connection pool but with no success.&lt;/p&gt;

&lt;p&gt;Our development team also tried different versions of the Java&apos;s mongo driver, also with no positive results. (currently using version 3.4.2)&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;com.mongodb.MongoExecutionTimeoutException: Operation timed out, request was RemoteCommand 254187776 -- target:mongo-shard-geral7a.foobar.com:7308 db:admin cmd:{ isMaster: 1 }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;com.mongodb.MongoExecutionTimeoutException: Couldn&apos;t get a connection within the time limit&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;We also tried to upgrade and downgrade the MongoDB binary versions (3.4.6, 3.4.3, 3.4.0) none of them had any effects to solve our problem.&lt;/p&gt;

&lt;p&gt;We have several individual replica-set deployments, and none of them faced this issue in the past three years.&lt;/p&gt;

&lt;p&gt;The data provided with this ticket is:&lt;br/&gt;
All MongoDB Nodes (from the given cluster) logs and diagnostic.data files (past 3 days).&lt;br/&gt;
All MongoDB Servers (from the given cluster) logs (past 3 days).&lt;br/&gt;
All MongoDB Configs (from the given cluster) logs (past 3 days).&lt;br/&gt;
Network tests between a &apos;mongos&apos;, a config server and a primary node.&lt;/p&gt;

&lt;p&gt;IP&#8217;s and DNS&#8217; has been hidden or modified for security reasons.&lt;/p&gt;</description>
                <environment></environment>
        <key id="415807">SERVER-30628</key>
            <summary>Timeouts between &apos;mongos&apos; and replicaset nodes rejecting requests</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="4">Incomplete</resolution>
                                        <assignee username="mark.agarunov">Mark Agarunov</assignee>
                                    <reporter username="lucasoares">Lucas</reporter>
                        <labels>
                    </labels>
                <created>Sat, 12 Aug 2017 02:50:15 +0000</created>
                <updated>Sun, 7 Jan 2018 04:36:28 +0000</updated>
                            <resolved>Fri, 8 Dec 2017 20:21:31 +0000</resolved>
                                    <version>3.4.0</version>
                    <version>3.4.3</version>
                    <version>3.4.4</version>
                    <version>3.4.6</version>
                                                    <component>Networking</component>
                    <component>Sharding</component>
                                        <votes>5</votes>
                                    <watches>18</watches>
                                                                                                                <comments>
                            <comment id="1747319" author="mark.agarunov" created="Fri, 8 Dec 2017 20:21:31 +0000"  >&lt;p&gt;Hello &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=lucasoares&quot; class=&quot;user-hover&quot; rel=&quot;lucasoares&quot;&gt;lucasoares&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;We haven&#8217;t heard back from you for some time, so I&#8217;m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information and we will reopen the ticket.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Mark&lt;/p&gt;</comment>
                            <comment id="1740341" author="mark.agarunov" created="Fri, 1 Dec 2017 18:52:08 +0000"  >&lt;p&gt;Hello &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=lucasoares&quot; class=&quot;user-hover&quot; rel=&quot;lucasoares&quot;&gt;lucasoares&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Is this still an issue for you are did the upgrade fix the behavior you were seeing?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Mark&lt;/p&gt;</comment>
                            <comment id="1722848" author="lucasoares" created="Fri, 10 Nov 2017 18:47:21 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mark.agarunov&quot; class=&quot;user-hover&quot; rel=&quot;mark.agarunov&quot;&gt;mark.agarunov&lt;/a&gt; important: we tried version 3.4.0 and 3.4.3 too. This &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-30643&quot; title=&quot;Performance regression with SSL&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-30643&quot;&gt;&lt;del&gt;SERVER-30643&lt;/del&gt;&lt;/a&gt; affects only 3.4.4+, correct?&lt;/p&gt;

&lt;p&gt;I really don&apos;t think this will complete solve the problem. But I will give a try.&lt;/p&gt;</comment>
                            <comment id="1722844" author="lucasoares" created="Fri, 10 Nov 2017 18:42:36 +0000"  >&lt;p&gt;Hello &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mark.agarunov&quot; class=&quot;user-hover&quot; rel=&quot;mark.agarunov&quot;&gt;mark.agarunov&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;I will upgrade both clusters in this weekend.&lt;/p&gt;

&lt;p&gt;A question: I still see, even without SSL, connections taking more than 500ms (sometimes more than 4 seconds). What to do about those cases?&lt;/p&gt;

&lt;p&gt;They are rare, but still occurs. Examples:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-11-10T18:04:22.768+0000 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-3-0] Successfully connected to mongo-shard-ibm8a:27017, took 5006ms (16 connections now open to mongo-shard-ibm8a:27017)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-11-10T18:04:22.774+0000 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-3-0] Successfully connected to mongo-shard-ibm8a:27017, took 4941ms (16 connections now open to mongo-shard-ibm8a:27017)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-11-10T18:04:22.776+0000 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-3-0] Successfully connected to mongo-shard-ibm8a:27017, took 4804ms (16 connections now open to mongo-shard-ibm8a:27017)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-11-10T18:04:22.777+0000 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-3-0] Successfully connected to mongo-shard-ibm8a:27017, took 4744ms (16 connections now open to mongo-shard-ibm8a:27017)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-11-10T18:10:56.362+0000 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-0-0] Successfully connected to mongo-shard-ibm9a:27017, took 5012ms (11 connections now open to mongo-shard-ibm9a:27017)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-11-10T18:10:56.372+0000 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-0-0] Successfully connected to mongo-shard-ibm9a:27017, took 4994ms (11 connections now open to mongo-shard-ibm9a:27017)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-11-10T17:12:23.392+0000 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-1-0] Successfully connected to mongo-shard-ibm7c:27017, took 5010ms (8 connections now open to mongo-shard-ibm7c:27017)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-11-10T18:29:05.534+0000 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-6-0] Successfully connected to mongo-shard-ibm9c:27017, took 5018ms (6 connections now open to mongo-shard-ibm9c:27017)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-11-10T18:31:48.835+0000 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-0-0] Successfully connected to mongo-shard-ibm5c:27017, took 5007ms (12 connections now open to mongo-shard-ibm5c:27017)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-11-10T16:33:01.118+0000 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-5-0] Successfully connected to mongo-shard-ibm6b:27017, took 5018ms (10 connections now open to mongo-shard-ibm6b:27017)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-11-10T18:10:56.364+0000 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-1-0] Successfully connected to mongo-shard-ibm9a:27017, took 5014ms (9 connections now open to mongo-shard-ibm9a:27017)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="1722799" author="mark.agarunov" created="Fri, 10 Nov 2017 18:08:44 +0000"  >&lt;p&gt;Hello &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=lucasoares&quot; class=&quot;user-hover&quot; rel=&quot;lucasoares&quot;&gt;lucasoares&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for providing this additional data. After further investigation, I suspect this behavior may be due to a bug related to SSL, as detailed in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-30643&quot; title=&quot;Performance regression with SSL&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-30643&quot;&gt;&lt;del&gt;SERVER-30643&lt;/del&gt;&lt;/a&gt;, which was fixed in 3.4.9. If possible, please upgrade to 3.4.9 or newer and see if this fixes the issue.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Mark &lt;/p&gt;</comment>
                            <comment id="1721824" author="lucasoares" created="Thu, 9 Nov 2017 18:47:31 +0000"  >&lt;p&gt;I saw those connections taking a long time to connect in the cluster mentioned in the description (still using preferSSL mode):&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-11-09T18:31:46.216+0000 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-2-0] Successfully connected to mongo-zeroshard4a:27017, took 3323ms (32 connections now open to mongo-zeroshard4a:27017)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-11-09T18:31:46.217+0000 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-2-0] Successfully connected to mongo-zeroshard4a:27017, took 3539ms (32 connections now open to mongo-zeroshard4a:27017)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-11-09T18:31:46.260+0000 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-0-0] Successfully connected to mongo-zeroshard4a:27017, took 2883ms (29 connections now open to mongo-zeroshard4a:27017)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-11-09T18:33:10.745+0000 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-2-0] Successfully connected to mongo-zeroshard4a:27017, took 938ms (29 connections now open to mongo-zeroshard4a:27017)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-11-09T18:33:10.745+0000 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-2-0] Successfully connected to mongo-zeroshard4a:27017, took 2821ms (29 connections now open to mongo-zeroshard4a:27017)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-11-09T18:33:10.752+0000 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-2-0] Successfully connected to mongo-zeroshard4a:27017, took 2469ms (29 connections now open to mongo-zeroshard4a:27017)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;I executed the command you ask in the &apos;mongo-zeroshard4a&apos; and uploaded the file &apos;mongo-zeroshard4a-gdb.txt&apos;.&lt;/p&gt;

&lt;p&gt;I also executed the command in one mongod from the cluster I mentioned in the comments (using allowSSL mode). I saw a connection taking 50 seconds to complete in the mongos log and I executed the command in the &apos;mongod&apos; target. I uploaded the file called &apos;mongo-shard-ibm8a-gdb.txt&apos;.&lt;/p&gt;

&lt;p&gt;In both mongod the command show this error while exeuting:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;185	../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such file or directory.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;185	../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such file or directory.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;185	../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such file or directory.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;185	../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such file or directory.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;185	../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such file or directory.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="1721808" author="lucasoares" created="Thu, 9 Nov 2017 18:31:29 +0000"  >&lt;p&gt;Hello &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mark.agarunov&quot; class=&quot;user-hover&quot; rel=&quot;mark.agarunov&quot;&gt;mark.agarunov&lt;/a&gt;. Since this cluster is a production one, we did a lot of things to find a workaround. One of those workarounds was disabling preferSSL and changing all cluster to allowSSL mode; then we changed our applications to connect in the MongoDB without ssl protocol. We did this two days ago, and we saw a huge decreased in the connection time.&lt;/p&gt;

&lt;p&gt;The command you asked to run will work even without preferSSL mode?&lt;/p&gt;

&lt;p&gt;And I&apos;m confused now. You asked to run the command in a &apos;mongod&apos;, right? What I can do is find the shard that is taking the most time to connect (seeing the mongos log) and execute the command there. Will this work?&lt;/p&gt;

&lt;p&gt;Another thing I can do is to execute this command in the other cluster (the one I mentioned in the description, not in the comment) and maybe we will get more information about the issue. In this cluster connections are taking a little bit longer to connect.&lt;/p&gt;

&lt;p&gt;For documentation, what I did so far searching for a workaround:&lt;/p&gt;

&lt;p&gt;1. Deployed one mongos per application (this starts to work but isn&apos;t a good, since we have more than 100 running applications).&lt;br/&gt;
2. Removed the ssl protocol from all applications connecting to the mongos.&lt;br/&gt;
3. Deployed 15 applications using only 3 mongos addresses to connect. I did this yesterday to find a solution to the workaround 1, since we can&apos;t manage those many mongos servers. It&apos;s working so far.&lt;br/&gt;
4. Limited the pool size of the Java driver to 35 connections. We did this to prevent connection pooling in the mongos. When we saw the connection time growing up to 40 seconds, we figure out the &apos;mongos&apos; are creating a whole bunch of connection (and killing them when they become idle). This was a infinite loop and limiting the pool size maybe prevents this happens.&lt;br/&gt;
5. Changed the read preference to secondary preferred. We know the production notes doesn&apos;t advise secondary reads, but so far this really improved our connections. We don&apos;t have any aggressive load to the primaries since they have SSD, lots of CPUs, RAM and a good network. I think the connection pooling with secondaries maybe prevent the &apos;mongos&apos; of creating lots of connections and stay with those connections already open.&lt;/p&gt;

&lt;p&gt;Sometimes we still see some connections taking 600ms~5s to connect, but in the most cases connections only take 5~20ms, like this:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;2017-11-09T17:49:58.314+0000 I ASIO     &lt;span class=&quot;error&quot;&gt;&amp;#91;NetworkInterfaceASIO-TaskExecutorPool-4-0&amp;#93;&lt;/span&gt; Successfully connected to mongo-shard-ibm2a:27017, took 9ms (6 connections now open to mongo-shard-ibm2a:27017)&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Those times when the connection needs 5 seconds to establish worries me a lot. I&apos;m trying to create a tcpdump to get some insights of it.&lt;/p&gt;

&lt;p&gt;We previously had 20 mongos, and the application always connects to all of them (using all addresses in java driver connector). We expect the driver balance do the work, but that&apos;s how it all started. It appeared that only a few of them were receiving operations in this case.&lt;/p&gt;</comment>
                            <comment id="1721741" author="mark.agarunov" created="Thu, 9 Nov 2017 17:42:03 +0000"  >&lt;p&gt;Hello &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=lucasoares&quot; class=&quot;user-hover&quot; rel=&quot;lucasoares&quot;&gt;lucasoares&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for the additional information. After looking over this some more, I suspect this may be due to an issue when using &lt;tt&gt;requireSSL&lt;/tt&gt; or &lt;tt&gt;preferSSL&lt;/tt&gt; mode. To confirm that this is the cause, could you please execute the following on an affected &lt;tt&gt;mongod&lt;/tt&gt; node when you are seeing this issue and provide the &lt;tt&gt;gdb.txt&lt;/tt&gt; file that is generated?&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;for i in $(seq 5); do gdb -ex &quot;thread apply all bt&quot; --batch --pid $(pidof mongod); date; sleep 10; done &amp;gt; gdb.txt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Mark&lt;/p&gt;</comment>
                            <comment id="1713635" author="lucasoares" created="Tue, 31 Oct 2017 16:22:18 +0000"  >&lt;p&gt;Hello again!&lt;/p&gt;

&lt;p&gt;I set a cuple parameters and then the &apos;mongos&apos; aren&apos;t exceeding timeout to connect to shards:&lt;br/&gt;
  cursorTimeoutMillis: &apos;1800000&apos;&lt;br/&gt;
  ShardingTaskExecutorPoolHostTimeoutMS: &apos;300000000&apos;&lt;br/&gt;
  ShardingTaskExecutorPoolRefreshTimeoutMS: &apos;600000&apos;&lt;/p&gt;

&lt;p&gt;Now I can see how long connections are taking to stabilish (around 10~40 seconds). &lt;/p&gt;

&lt;p&gt;I don&apos;t know why, since all my servers have a 10Gbps network and average ping of 0.02ms. Manually connecting to the shards are really fast.&lt;/p&gt;

&lt;p&gt;My mongos have an average of 2k connections, and my shards an average of 4k.&lt;/p&gt;

&lt;p&gt;I will keep diagnosing. Thanks!&lt;/p&gt;</comment>
                            <comment id="1713085" author="lucasoares" created="Mon, 30 Oct 2017 23:18:27 +0000"  >&lt;p&gt;Using glibc 2.23 doesn&apos;t solved the problem.&lt;/p&gt;</comment>
                            <comment id="1712636" author="lucasoares" created="Mon, 30 Oct 2017 16:52:30 +0000"  >&lt;p&gt;Hello.&lt;/p&gt;

&lt;p&gt;I&apos;m investigating older issues in the same component and I found interesting things.&lt;/p&gt;

&lt;p&gt;First of all, I&apos;m using RHEL 7.3 (kernel 3.10 and glibc 2.17) on my mongo servers. The issue &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26723&quot; title=&quot;Mongos stalls for even simple queries&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-26723&quot;&gt;&lt;del&gt;SERVER-26723&lt;/del&gt;&lt;/a&gt; says, and I quote:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;We have completed our investigation and concluded that the issue described in this ticket is caused by a bug in glibc.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;The issue &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26723&quot; title=&quot;Mongos stalls for even simple queries&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-26723&quot;&gt;&lt;del&gt;SERVER-26723&lt;/del&gt;&lt;/a&gt; doesn&apos;t have the same error pattern of this issue, but I can&apos;t ignore and I will install 5 new mongo servers with one SO with kernel v4 and then I will try to replicate the same problem.&lt;/p&gt;

&lt;p&gt;I think issues &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26654&quot; title=&quot;ExceededTimeLimit: Operation timed out after uprgade from 3.2.8 to 3.2.10&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-26654&quot;&gt;&lt;del&gt;SERVER-26654&lt;/del&gt;&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-29206&quot; title=&quot;MongoS [NetworkInterfaceASIO-TaskExecutorPool-0-0] ExceededTimeLimit: Operation timed out&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-29206&quot;&gt;&lt;del&gt;SERVER-29206&lt;/del&gt;&lt;/a&gt; have more likely to be the same problem as I have, but with different mongo verions. One of the reporters also have a different operating system.&lt;/p&gt;</comment>
                            <comment id="1709613" author="lucasoares" created="Wed, 25 Oct 2017 23:31:20 +0000"  >&lt;p&gt;OK. Please, ask anything you need to debug this. I&apos;m here to help because I really want this problem fixed or a good explanation to help me to solve this.&lt;/p&gt;

&lt;p&gt;Thanks.&lt;/p&gt;</comment>
                            <comment id="1708896" author="mark.agarunov" created="Wed, 25 Oct 2017 17:20:57 +0000"  >&lt;p&gt;Hello &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=lucasoares&quot; class=&quot;user-hover&quot; rel=&quot;lucasoares&quot;&gt;lucasoares&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for providing the additional information. Unfortunately we have not yet determined the cause of this behavior and are still investigating. However the log you provided should give some additional insight into this.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Mark&lt;/p&gt;</comment>
                            <comment id="1708167" author="lucasoares" created="Wed, 25 Oct 2017 03:12:12 +0000"  >&lt;p&gt;I uploaded a log file with verbose level 5 of one mongos with this same issue. I hope this helps. The file name is &apos;logging_verbose_5.7z&apos;.&lt;/p&gt;</comment>
                            <comment id="1707739" author="lucasoares" created="Tue, 24 Oct 2017 18:54:02 +0000"  >&lt;p&gt;Hello &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mark.agarunov&quot; class=&quot;user-hover&quot; rel=&quot;mark.agarunov&quot;&gt;mark.agarunov&lt;/a&gt;. I&apos;m still having this issue but now with a brand new cluster from a different datacenter with a different server provider. All my team believe the problem with MongoDB, because we could not find a solution or a functional workaround.&lt;/p&gt;

&lt;p&gt;We are trying to add lot of &apos;mongos&apos; instances, and some of those are beeing used by one application of ours. This minimized the problem but is not a good solution to us, since we have more than 300 application servers running.&lt;/p&gt;

&lt;p&gt;We tried to use the same NTP server in all our servers as advised by one of MongoDB internal employee.&lt;/p&gt;

&lt;p&gt;We tried to change lots of infrastructure configuration on our servers following the production notes of MongoDB.&lt;/p&gt;

&lt;p&gt;We tried lot of things, and nothing are helping.&lt;/p&gt;

&lt;p&gt;Maybe the problem is due to an infrastructure configuration, but even different versions of OS we tried (Ubuntu 14.04 and 16.04). Unfortunately I&apos;m not having any success to prove that isn&apos;t a MongoDB issue. I really need help at least saying what kind of parameter or configuration I can try to change, or at least one update of this issue.&lt;/p&gt;

&lt;p&gt;Thanks!&lt;/p&gt;</comment>
                            <comment id="1680016" author="mark.agarunov" created="Fri, 22 Sep 2017 21:21:27 +0000"  >&lt;p&gt;Hello &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=lucasoares&quot; class=&quot;user-hover&quot; rel=&quot;lucasoares&quot;&gt;lucasoares&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for providing these files. My apologies for the delay in response, unfortunately we have not yet determined the cause of this behavior, but we are still investigating this issue. We will provide updates on this as they&apos;re available.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Mark&lt;/p&gt;</comment>
                            <comment id="1648161" author="lucasoares" created="Mon, 14 Aug 2017 22:49:07 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ramon.fernandez&quot; class=&quot;user-hover&quot; rel=&quot;ramon.fernandez&quot;&gt;ramon.fernandez&lt;/a&gt;. We uploaded the files.&lt;/p&gt;</comment>
                            <comment id="1647404" author="ramon.fernandez" created="Mon, 14 Aug 2017 12:34:33 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=lucasoares&quot; class=&quot;user-hover&quot; rel=&quot;lucasoares&quot;&gt;lucasoares&lt;/a&gt;, I&apos;ve created an &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/ce9a7a26-3c20-4a62-9abb-a302cba909b3.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;upload portal&lt;/a&gt; so you can send logs and diagnostic data; please let us know when the uploads are complete.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="1646810" author="lucasoares" created="Sat, 12 Aug 2017 02:52:11 +0000"  >&lt;p&gt;Unfortunately there is a limit of 150MB for attachments. We have tons of logs and diagnostic.data files from past 3 days to upload.&lt;/p&gt;

&lt;p&gt;Can you guys open a private portal to upload these files?&lt;/p&gt;

&lt;p&gt;Thanks.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="416320">SERVER-30643</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="167395" name="connections.png" size="258461" author="mark.agarunov" created="Thu, 5 Oct 2017 18:59:43 +0000"/>
                            <attachment id="167390" name="ftdc.png" size="141347" author="mark.agarunov" created="Thu, 5 Oct 2017 18:28:22 +0000"/>
                            <attachment id="170592" name="mongo-shard-ibm8a-gdb.txt" size="12837907" author="lucasoares" created="Thu, 9 Nov 2017 18:49:13 +0000"/>
                            <attachment id="170593" name="mongo-zeroshard4a-gdb.txt" size="23712530" author="lucasoares" created="Thu, 9 Nov 2017 18:50:06 +0000"/>
                            <attachment id="163166" name="reportNetwork_mongo-20170811.txt" size="17053" author="lucasoares" created="Sat, 12 Aug 2017 00:07:31 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>19.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 14 Aug 2017 12:34:33 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        6 years, 9 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>backlog-server-pm</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            6 years, 9 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>lucasoares</customfieldvalue>
            <customfieldvalue>mark.agarunov</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htcvmn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|ht4msn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htchpj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>