<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:13:01 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-26723] Mongos stalls for even simple queries</title>
                <link>https://jira.mongodb.org/browse/SERVER-26723</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We have a 32 node cluster where each node is a 3 member replicaset having a hidden member.&lt;br/&gt;
Cluster is running MongoDb 3.2.5 with WiredTiger on RHEL7.&lt;br/&gt;
We see that some mongos processes get stuck and just do not return even for a very simple query while most other mongos processes are not affected.&lt;br/&gt;
This happens to different nodes and is not limited to specific machines. It had happened yesterday on another machine and then after restarting the cluster, it happened on this machine.&lt;/p&gt;

&lt;p&gt;Queries like count() or show dbs or show collections that most probably access only the metadata - work fine on the mongos of the affected machine. However, running any simple query stalls the mongos.&lt;/p&gt;

&lt;p&gt;Attaching mongos logs for a mongos process that is not responding as well the config server primary node log.&lt;br/&gt;
Note that the mongo processes were started with -v flag to increase logging verbosity&lt;/p&gt;</description>
                <environment></environment>
        <key id="325869">SERVER-26723</key>
            <summary>Mongos stalls for even simple queries</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="kaloian.manassiev@mongodb.com">Kaloian Manassiev</assignee>
                                    <reporter username="darshan.shah@interactivedata.com">Darshan Shah</reporter>
                        <labels>
                    </labels>
                <created>Fri, 21 Oct 2016 14:10:33 +0000</created>
                <updated>Thu, 1 Feb 2018 15:21:10 +0000</updated>
                            <resolved>Mon, 28 Nov 2016 23:30:59 +0000</resolved>
                                    <version>3.2.5</version>
                                                    <component>Sharding</component>
                                        <votes>1</votes>
                                    <watches>18</watches>
                                                                                                                <comments>
                            <comment id="1791777" author="darshan.shah@interactivedata.com" created="Thu, 1 Feb 2018 15:21:10 +0000"  >&lt;p&gt;Just FYI - the original problem was resolved when we upgraded MongoDb to 3.2.11 and also upgraded glibc to 2.17.&lt;/p&gt;

&lt;p&gt;However now we are noticing problems similar to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26722&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;SERVER-26722&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-29206&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;SERVER-29206&lt;/a&gt; even after upgrading to MongoDb 3.2.18 running on RHEL7.1.&lt;/p&gt;</comment>
                            <comment id="1446944" author="vick.mena" created="Thu, 1 Dec 2016 16:02:23 +0000"  >&lt;p&gt;To Summarize:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Users running glibc versions &amp;lt; 2.21 are susceptible to the infinite loop issue&lt;/li&gt;
	&lt;li&gt;Users running glibc versions &amp;lt; 2.23 are susceptible to the kernel bugs in 3.17-3.19&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Recommendations:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Upgrade glibc to a minimum 2.21&lt;/li&gt;
	&lt;li&gt;Users running the affected kernels should upgrade to the latest v4.0 for their vendor and optionally upgrade glibc to 2.23&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;As a reminder, RHEL 7.3 has kernel 3.10 and glibc 2.17&lt;/p&gt;</comment>
                            <comment id="1444273" author="royrez@microsoft.com" created="Tue, 29 Nov 2016 08:37:10 +0000"  >&lt;p&gt;The bug that you linked to is still in the state &quot;NEW&quot; meaning that there is no operating system that contains a fix for that.&lt;br/&gt;
Can you specify which kernel versions are safe to use?&lt;/p&gt;</comment>
                            <comment id="1444068" author="thomas.schubert" created="Mon, 28 Nov 2016 23:30:59 +0000"  >&lt;p&gt;Hi all,&lt;/p&gt;

&lt;p&gt;We have completed our investigation and concluded that the issue described in this ticket is caused by a bug in glibc. Therefore, we will be resolving this ticket, as there is no work to be done on the MongoDB server to correct this behavior. &lt;/p&gt;

&lt;p&gt;To ensure that you are not affected by this issue, please upgrade your operating system to a version that contains the fix for &lt;a href=&quot;https://bugzilla.kernel.org/show_bug.cgi?id=99671&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://bugzilla.kernel.org/show_bug.cgi?id=99671&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Please be aware that this issue is distinct from &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26722&quot; title=&quot;router blocks and throws ExceededTimeLimit&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-26722&quot;&gt;&lt;del&gt;SERVER-26722&lt;/del&gt;&lt;/a&gt;, which is not caused by a bug in glibc. We are continuing to investigate this issue, and will update &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26722&quot; title=&quot;router blocks and throws ExceededTimeLimit&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-26722&quot;&gt;&lt;del&gt;SERVER-26722&lt;/del&gt;&lt;/a&gt; when we know more.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1442819" author="royrez@microsoft.com" created="Sun, 27 Nov 2016 16:46:22 +0000"  >&lt;p&gt;Any news on this?&lt;br/&gt;
It&apos;s a ticket causing a complete mongo meltdown on sharded environments and logs were provided a while ago already.&lt;/p&gt;</comment>
                            <comment id="1440252" author="thomas.schubert" created="Tue, 22 Nov 2016 17:04:06 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=alessandro.gherardi%40yahoo.com&quot; class=&quot;user-hover&quot; rel=&quot;alessandro.gherardi@yahoo.com&quot;&gt;alessandro.gherardi@yahoo.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thanks for uploading the logs. We are still investigating this issue and will let update this ticket when we know more.&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1435933" author="alessandro.gherardi@yahoo.com" created="Wed, 16 Nov 2016 20:09:54 +0000"  >&lt;p&gt;@Kaloian Manassiev - I uploaded log files from our server to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26654&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-26654&lt;/a&gt; . I understand that ticket is now closed - feel free to move those log files here.&lt;/p&gt;</comment>
                            <comment id="1435915" author="ramon.fernandez" created="Wed, 16 Nov 2016 19:55:18 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jon%40appboy.com&quot; class=&quot;user-hover&quot; rel=&quot;jon@appboy.com&quot;&gt;jon@appboy.com&lt;/a&gt;, unfortunately we have not yet been able to confirm the root cause for this issue, so it can&apos;t be part of 3.2.11.&lt;/p&gt;

&lt;p&gt;As &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kaloian.manassiev&quot; class=&quot;user-hover&quot; rel=&quot;kaloian.manassiev&quot;&gt;kaloian.manassiev&lt;/a&gt; mentioned above we&apos;re trying to determine if the problem is related to the &lt;tt&gt;glibc&lt;/tt&gt; bug listed above.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="1435491" author="jonhyman" created="Wed, 16 Nov 2016 13:13:00 +0000"  >&lt;p&gt;I noticed that the 3.2.11-rc1 note came out today - is this going to be targeted for 3.2.11? We are unable to upgrade because of this issue / &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26654&quot; title=&quot;ExceededTimeLimit: Operation timed out after uprgade from 3.2.8 to 3.2.10&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-26654&quot;&gt;&lt;del&gt;SERVER-26654&lt;/del&gt;&lt;/a&gt;, so would appreciate if this could be attempted to make it into 3.2.11.&lt;/p&gt;</comment>
                            <comment id="1433833" author="darshan.shah@interactivedata.com" created="Mon, 14 Nov 2016 22:15:03 +0000"  >&lt;p&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;oasprod@bxb-ppe-oas002 tmp&amp;#93;&lt;/span&gt;$ uname -a&lt;br/&gt;
Linux bxb-ppe-oas002 3.10.0-229.el7.x86_64 #1 SMP Thu Jan 29 18:37:38 EST 2015 x86_64 x86_64 x86_64 GNU/Linux&lt;/p&gt;</comment>
                            <comment id="1432067" author="kaloian.manassiev" created="Fri, 11 Nov 2016 16:47:49 +0000"  >&lt;p&gt;Thanks a lot, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=darshan.shah%40interactivedata.com&quot; class=&quot;user-hover&quot; rel=&quot;darshan.shah@interactivedata.com&quot;&gt;darshan.shah@interactivedata.com&lt;/a&gt; for providing the complete thread stack traces. Based on this information I see that all but one of the asio host name resolution threads are stuck in the hostname resolution call:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#0  0x00007f4f676fdb6c in __lll_lock_wait_private () from /lib64/libc.so.6&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#1  0x00007f4f67713efd in _L_lock_746 () from /lib64/libc.so.6&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#2  0x00007f4f67713cb5 in __check_pf () from /lib64/libc.so.6&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#3  0x00007f4f676d9d31 in getaddrinfo () from /lib64/libc.so.6&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;and one thread has not returned from a recvmsg call:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#0  0x00007f4f676f111d in recvmsg () from /lib64/libc.so.6&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#1  0x00007f4f6771390d in make_request () from /lib64/libc.so.6&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#2  0x00007f4f67713e04 in __check_pf () from /lib64/libc.so.6&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#3  0x00007f4f676d9d31 in getaddrinfo () from /lib64/libc.so.6&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#4  0x0000000000c8a9b7 in asio::detail::socket_ops::getaddrinfo(char const*, char const*, addrinfo const&amp;amp;, addrinfo**, std::error_code&amp;amp;) ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;This seems to be similar to a deadlock bug reported for glibc and we are now working on confirming that:&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;&lt;a href=&quot;https://bugzilla.redhat.com/show_bug.cgi?id=1209433&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://bugzilla.redhat.com/show_bug.cgi?id=1209433&lt;/a&gt;&lt;/li&gt;
	&lt;li&gt;&lt;a href=&quot;https://bugzilla.kernel.org/show_bug.cgi?id=99671&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://bugzilla.kernel.org/show_bug.cgi?id=99671&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Can you please run &quot;&lt;tt&gt;uname -a&lt;/tt&gt;&quot; on one of the mongos hosts exhibiting the problem and paste the output in this ticket so we can see which kernel you are running?&lt;/p&gt;

&lt;p&gt;Thanks in advance.&lt;/p&gt;

&lt;p&gt;-Kal.&lt;/p&gt;</comment>
                            <comment id="1426182" author="darshan.shah@interactivedata.com" created="Fri, 4 Nov 2016 13:22:41 +0000"  >&lt;p&gt;Unfortunately, I do not have the full thread dump from that particular date.&lt;br/&gt;
Attaching the full thread dump &amp;amp; the mongos log file from today from another server where I see this same problem.&lt;br/&gt;
Note that network connectivity between the problem server and other servers  is good - I can ping and ssh in both directions.&lt;/p&gt;

&lt;p&gt;Let me know if you need any other info.&lt;/p&gt;

&lt;p&gt;Thanks.&lt;/p&gt;</comment>
                            <comment id="1422994" author="kaloian.manassiev" created="Tue, 1 Nov 2016 15:37:39 +0000"  >&lt;p&gt;Thank you &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=darshan.shah%40interactivedata.com&quot; class=&quot;user-hover&quot; rel=&quot;darshan.shah@interactivedata.com&quot;&gt;darshan.shah@interactivedata.com&lt;/a&gt; for providing us with the call stacks - this is really helpful.&lt;/p&gt;

&lt;p&gt;The two stacks show that there are asynchronous networking threads which are stuck doing hostname resolution and are blocked in the OS. However from them I am unable to see whether all the client threads are also blocked on this hostname resolution or on something else.&lt;/p&gt;

&lt;p&gt;Would it be possible to attach all the threads&apos; stacks if you still have them?&lt;/p&gt;

&lt;p&gt;Alternatively you can use this command, which will output them to a file:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;gdb -ex &quot;thread apply all bt&quot; -p $(pidof mongos) &amp;gt; mongos.stacks&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Thank you in advance.&lt;/p&gt;

&lt;p&gt;-Kal.&lt;/p&gt;</comment>
                            <comment id="1418039" author="darshan.shah@interactivedata.com" created="Wed, 26 Oct 2016 13:40:56 +0000"  >&lt;p&gt;Here is the server info:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;$&amp;gt; uname -a&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Linux bxb-ppe-oas013 3.10.0-229.el7.x86_64 #1 SMP Thu Jan 29 18:37:38 EST 2015 x86_64 x86_64 x86_64 GNU/Linux&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;$&amp;gt; cat /etc/redhat-release&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Red Hat Enterprise Linux Server release 7.1 (Maipo)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;We observed that the number of threads in a stuck mongos process are way higher than on the one that&apos;s working fine.&lt;br/&gt;
Here is the stack trace from gdb on a stuck mongos process:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Thread 314 (Thread 0x7f2053ead700 (LWP 12953)):&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#0  0x00007f20872dc11d in recvmsg () from /lib64/libc.so.6&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#1  0x00007f20872fe90d in make_request () from /lib64/libc.so.6&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#2  0x00007f20872fee04 in __check_pf () from /lib64/libc.so.6&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#3  0x00007f20872c4d31 in getaddrinfo () from /lib64/libc.so.6&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#4  0x0000000000c8a9b7 in asio::detail::socket_ops::getaddrinfo(char const*, char const*, addrinfo const&amp;amp;, addrinfo**, std::error_code&amp;amp;) ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#5  0x0000000000c8aae3 in asio::detail::socket_ops::background_getaddrinfo(std::weak_ptr&amp;lt;void&amp;gt; const&amp;amp;, char const*, char const*, addrinfo const&amp;amp;, addrinfo**, std::error_code&amp;amp;) ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#6  0x0000000000a221c1 in asio::detail::resolve_op&amp;lt;asio::ip::tcp, asio::detail::wrapped_handler&amp;lt;asio::io_service::strand, mongo::executor::NetworkInterfaceASIO::_connect(mongo::executor::NetworkInterfaceASIO::AsyncOp*)::{lambda(std::error_code, asio::ip::basic_resolver_iterator&amp;lt;asio::ip::tcp&amp;gt;)#1}, asio::detail::is_continuation_if_running&amp;gt; &amp;gt;::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&amp;amp;, unsigned long) ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#7  0x0000000000c8c3e1 in asio::detail::scheduler::do_run_one(asio::detail::scoped_lock&amp;lt;asio::detail::posix_mutex&amp;gt;&amp;amp;, asio::detail::scheduler_thread_info&amp;amp;, std::error_code const&amp;amp;) ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#8  0x0000000000c8c601 in asio::detail::scheduler::run(std::error_code&amp;amp;) ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#9  0x0000000000c9079f in asio::io_service::run() ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#10 0x0000000000c83f40 in asio_detail_posix_thread_function ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;---Type &amp;lt;return&amp;gt; to continue, or q &amp;lt;return&amp;gt; to quit---&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#11 0x00007f20875addf5 in start_thread () from /lib64/libpthread.so.0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#12 0x00007f20872db1ad in clone () from /lib64/libc.so.6&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Thread 313 (Thread 0x7f20536ac700 (LWP 12957)):&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#0  0x00007f20872e8b6c in __lll_lock_wait_private () from /lib64/libc.so.6&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#1  0x00007f20872feefd in _L_lock_746 () from /lib64/libc.so.6&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#2  0x00007f20872fecb5 in __check_pf () from /lib64/libc.so.6&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#3  0x00007f20872c4d31 in getaddrinfo () from /lib64/libc.so.6&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#4  0x0000000000c8a9b7 in asio::detail::socket_ops::getaddrinfo(char const*, char const*, addrinfo const&amp;amp;, addrinfo**, std::error_code&amp;amp;) ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#5  0x0000000000c8aae3 in asio::detail::socket_ops::background_getaddrinfo(std::weak_ptr&amp;lt;void&amp;gt; const&amp;amp;, char const*, char const*, addrinfo const&amp;amp;, addrinfo**, std::error_code&amp;amp;) ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#6  0x0000000000a221c1 in asio::detail::resolve_op&amp;lt;asio::ip::tcp, asio::detail::wrapped_handler&amp;lt;asio::io_service::strand, mongo::executor::NetworkInterfaceASIO::_connect(mongo::executor::NetworkInterfaceASIO::AsyncOp*)::{lambda(std::error_code, asio::ip::basic_resolver_iterator&amp;lt;asio::ip::tcp&amp;gt;)#1}, asio::detail::is_continuation_if_running&amp;gt; &amp;gt;::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&amp;amp;, unsigned long) ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#7  0x0000000000c8c3e1 in asio::detail::scheduler::do_run_one(asio::detail::scoped_lock&amp;lt;asio::detail::posix_mutex&amp;gt;&amp;amp;, asio::detail::scheduler_thread_info&amp;amp;, std::error_code const&amp;amp;) ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#8  0x0000000000c8c601 in asio::detail::scheduler::run(std::error_code&amp;amp;) ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#9  0x0000000000c9079f in asio::io_service::run() ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#10 0x0000000000c83f40 in asio_detail_posix_thread_function ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#11 0x00007f20875addf5 in start_thread () from /lib64/libpthread.so.0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#12 0x00007f20872db1ad in clone () from /lib64/libc.so.6&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="1414841" author="darshan.shah@interactivedata.com" created="Fri, 21 Oct 2016 20:44:50 +0000"  >&lt;p&gt;After restarting the problematic mongos, it works fine.&lt;/p&gt;

&lt;p&gt;In the stalled mongos logs, I see it tries to connect to chi-ppe-oas009 on port 29111 which is the primary mongod for that replicaset.&lt;/p&gt;

&lt;p&gt;So attaching the full mongod log for chi-ppe-oas009 and the log for the stalled mongos (on bxb-ppe-oas010) after the restart.&lt;/p&gt;</comment>
                            <comment id="1414793" author="ramon.fernandez" created="Fri, 21 Oct 2016 19:49:21 +0000"  >&lt;p&gt;Thanks for the report, we&apos;ll take a look at the logs for clues.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="324096">SERVER-26654</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="325251">SERVER-26701</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="325826">SERVER-26722</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="143361" name="bxb-ppe-oas006-mongos.stacks" size="2498839" author="darshan.shah@interactivedata.com" created="Fri, 4 Nov 2016 13:22:41 +0000"/>
                            <attachment id="142109" name="mongod.log-20161021-chi-ppe-oas009.gz" size="65598590" author="darshan.shah@interactivedata.com" created="Fri, 21 Oct 2016 20:44:50 +0000"/>
                            <attachment id="142078" name="mongodb_configdb-20161021-bxb-ppe-oas002.log.gz" size="137870853" author="darshan.shah@interactivedata.com" created="Fri, 21 Oct 2016 14:10:33 +0000"/>
                            <attachment id="142110" name="mongos.log-20161021-bxb-ppe-oas010-after_restart.gz" size="12410" author="darshan.shah@interactivedata.com" created="Fri, 21 Oct 2016 20:44:50 +0000"/>
                            <attachment id="142079" name="mongos.log-20161021-bxb-ppe-oas010.gz" size="108143876" author="darshan.shah@interactivedata.com" created="Fri, 21 Oct 2016 14:10:33 +0000"/>
                            <attachment id="143360" name="mongos_bxb-ppe-oas006.log" size="2402362" author="darshan.shah@interactivedata.com" created="Fri, 4 Nov 2016 13:22:41 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>16.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[500A000000VmTHNIA3]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 21 Oct 2016 14:32:03 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        6 years, 1 week, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>darshan.shah@interactivedata.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            6 years, 1 week, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>alessandro.gherardi@yahoo.com</customfieldvalue>
            <customfieldvalue>darshan.shah@interactivedata.com</customfieldvalue>
            <customfieldvalue>jonhyman</customfieldvalue>
            <customfieldvalue>kaloian.manassiev@mongodb.com</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
            <customfieldvalue>royrez@microsoft.com</customfieldvalue>
            <customfieldvalue>vick.mena</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrjswf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hsr2xj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrlhjb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>