<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:31:26 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-13352] Socket exception (SEND_ERROR) even after SERVER-9022 applied</title>
                <link>https://jira.mongodb.org/browse/SERVER-13352</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;div class=&quot;panel&quot; style=&quot;background-color: #EEEEEE;border-color: #ccc;border-width: 1px;&quot;&gt;&lt;div class=&quot;panelHeader&quot; style=&quot;border-bottom-width: 1px;border-bottom-color: #ccc;background-color: #6CB33F;&quot;&gt;&lt;b&gt;Issue Status as of Jan 09, 2015&lt;/b&gt;&lt;/div&gt;&lt;div class=&quot;panelContent&quot; style=&quot;background-color: #EEEEEE;&quot;&gt;
&lt;p&gt;&lt;b&gt;ISSUE SUMMARY&lt;/b&gt;&lt;br/&gt;
Certain network settings and/or events may cause the connection pools used by MongoDB to be populated by &quot;bad&quot; or &quot;broken&quot; connections. Common causes included periodic network failures and firewalls silently killing long running connections, though the actual cause was sometimes impossible to ascertain.&lt;/p&gt;

&lt;p&gt;These connections only reveal themselves to be unusable when they are selected from the pool and data is written to them, prior to that they appear to be healthy and usable.  This is particularly relevant to large sharded clusters which contain many connection pools (each mongos process and each primary for a shard have connection pools that can be impacted).&lt;/p&gt;

&lt;p&gt;&lt;b&gt;USER IMPACT&lt;/b&gt;&lt;br/&gt;
When a triggering event occurs, some proportion of the idle connections in a connection pool may become unusable, but still look healthy. Over time, as the MongoDB process (mongod or mongos) attempts to use these connections from the pool they may fail, throwing socket exceptions (SEND_ERROR, recv() timeout etc.).  These errors occur sporadically (depending on how many connections were affected, and how busy the process was) until such time as the &quot;bad&quot; connections in the pool are exhausted, or the process in question is restarted.  Essentially, this often presents as seemingly random socket exceptions long after the trigger event had occurred.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;WORKAROUNDS&lt;/b&gt;&lt;br/&gt;
If there is a suspected regular trigger event occurring then preventing the event in the first place is the best solution. If that proves elusive, the only definitive solution is to restart the impacted processes once such an event has occurred (or is suspected to have occurred) in order to clear out the problematic pools.&lt;/p&gt;

&lt;p&gt;The releaseConnectionsAfterResponse parameter (added in 2.2.4 and 2.4.2 as part of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-9022&quot; title=&quot;Enable mongos mode which releases ShardConnections to the pool after read ops&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-9022&quot;&gt;&lt;del&gt;SERVER-9022&lt;/del&gt;&lt;/a&gt;) can help alleviate the issue, but does not eliminate it.  Additionally, this parameter must be used judiciously and with caution, per the warning given in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-9022&quot; title=&quot;Enable mongos mode which releases ShardConnections to the pool after read ops&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-9022&quot;&gt;&lt;del&gt;SERVER-9022&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;AFFECTED VERSIONS&lt;/b&gt;&lt;br/&gt;
MongoDB versions prior to 2.6.0 are affected by this issue.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;FIX VERSION&lt;/b&gt;&lt;br/&gt;
The fix is included in the 2.6.0 production release.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;RESOLUTION DETAILS&lt;/b&gt;&lt;br/&gt;
MongoDB 2.6 comes with a new connection pooling code that includes the work done in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-9041&quot; title=&quot;proactively detect broken connections detected by the network&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-9041&quot;&gt;&lt;del&gt;SERVER-9041&lt;/del&gt;&lt;/a&gt; to proactively detect the re-use of broken connections from the pool.&lt;/p&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;h6&gt;&lt;a name=&quot;Originaldescription&quot;&gt;&lt;/a&gt;Original description&lt;/h6&gt;

&lt;p&gt;Like some other folks I was encountering the issue described in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-7008&quot; title=&quot;socket exception [SEND_ERROR] on Mongo Sharding&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-7008&quot;&gt;&lt;del&gt;SERVER-7008&lt;/del&gt;&lt;/a&gt; (principally on a cluster with 32 mongos, and 20 mongod forming 10 shards, all running 2.4.9).&lt;/p&gt;

&lt;p&gt;The occurrences were a bit random but tended to occur in the mornings and tended to occur early in the week (the latter probably correlated with weekly compaction that occurs on sat night). &lt;/p&gt;

&lt;p&gt;The problem would always disappear for 1-2 weeks after a mongos restart.&lt;/p&gt;

&lt;p&gt;After applying &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-9022&quot; title=&quot;Enable mongos mode which releases ShardConnections to the pool after read ops&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-9022&quot;&gt;&lt;del&gt;SERVER-9022&lt;/del&gt;&lt;/a&gt;, the problems had appeared to have stopped. After ~6 weeks some nodes started to see SEND_ERROR exceptions however. As before a mongos restart fixed everything.&lt;/p&gt;

&lt;p&gt;I confirmed that all the servers did have the patch applied (was: true)&lt;/p&gt;</description>
                <environment></environment>
        <key id="127616">SERVER-13352</key>
            <summary>Socket exception (SEND_ERROR) even after SERVER-9022 applied</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="ramon.fernandez@mongodb.com">Ramon Fernandez Marina</assignee>
                                    <reporter username="apiggott@ikanow.com">Alex Piggott</reporter>
                        <labels>
                    </labels>
                <created>Wed, 26 Mar 2014 02:22:43 +0000</created>
                <updated>Mon, 12 Jan 2015 21:39:24 +0000</updated>
                            <resolved>Mon, 12 Jan 2015 21:23:08 +0000</resolved>
                                    <version>2.4.9</version>
                                    <fixVersion>2.6.0</fixVersion>
                                    <component>Networking</component>
                    <component>Sharding</component>
                                        <votes>6</votes>
                                    <watches>15</watches>
                                                                                                                <comments>
                            <comment id="800841" author="apiggott@ikanow.com" created="Fri, 9 Jan 2015 03:36:15 +0000"  >&lt;p&gt;(I&apos;m the original reporter for this, we saw them regularly on a large &lt;del&gt;~20 node&lt;/del&gt; cluster and less frequently on smaller clusters, running 2.2 and 2.4)&lt;/p&gt;

&lt;p&gt;Since moving to mongodb 2.6 we have stopped seeing these problems&lt;/p&gt;</comment>
                            <comment id="679816" author="sflint@sailthru.com" created="Sun, 3 Aug 2014 03:39:39 +0000"  >&lt;p&gt;I am seeing issues as well.   This started as I added in 5th mongod to our cluster on each shard.   Currently on 2.4.8 in production. &lt;/p&gt;</comment>
                            <comment id="675773" author="srinivasa_kanamatha@mcafee.com" created="Tue, 29 Jul 2014 23:26:22 +0000"  >&lt;p&gt;We are seeing similar issues in our environment. Current version we are on 2.4.5. &lt;/p&gt;

&lt;p&gt;Here is the error&lt;/p&gt;

&lt;p&gt;&quot;socket exception &lt;span class=&quot;error&quot;&gt;&amp;#91;SEND_ERROR&amp;#93;&lt;/span&gt; for  ...&quot;&lt;/p&gt;</comment>
                            <comment id="674661" author="adamc" created="Tue, 29 Jul 2014 08:43:00 +0000"  >&lt;p&gt;Hi J&#233;r&#233;mie,&lt;/p&gt;

&lt;p&gt;I am the one responsible for writing and recording M202 and I just wanted to clarify the course content regarding connection pooling.  I do indeed talk about the need to restart the mongos in certain scenarios, however that section of the course actually states that 2.6 has made it less likely that you need to restart the mongos, so I think there is something of a misunderstanding.  When I speak about the need to restart I am referring to versions 2.4 (and below) because there have been specific improvements made in 2.6 around how to handle connections in the pool that have &quot;gone bad&quot;.  Therefore, the upgrade to 2.6 is definitely a recommended action for people seeing this problem, because (as mentioned in the course) the changes are too complex to backport to 2.4.&lt;/p&gt;

&lt;p&gt;Adam&lt;/p&gt;</comment>
                            <comment id="650873" author="apiggott@ikanow.com" created="Mon, 7 Jul 2014 20:26:25 +0000"  >&lt;p&gt;@Thomas: sorry I haven&apos;t had a chance to test it (the change in package structure for 2.6 has messed up our deployment infrastructure, obviously not a big issue to update but does mean I&apos;ve been waiting for someone to have some free time to do it, vs just being able to update one of our reference instances in 5 minutes)&lt;/p&gt;</comment>
                            <comment id="650863" author="jcharest" created="Mon, 7 Jul 2014 20:19:25 +0000"  >&lt;p&gt;I took the &quot;M202: MongoDB Advanced Deployment and Operations&quot; course recently on MongoDB University. At some point during the course, one presentation illustrated mongos connection pooling and clearly state that under mongo 2.6  we have to restart the mongos if that kind of network/socket error occur (near 7min in the video).&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=v8IGPu8XLKo&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://www.youtube.com/watch?v=v8IGPu8XLKo&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Will it be fix in 2.4.x or we absolutely need to uprade to 2.6? I think it might be a good to mention that information in the mongos documentation.&lt;/p&gt;</comment>
                            <comment id="650821" author="thomasr" created="Mon, 7 Jul 2014 19:38:32 +0000"  >&lt;p&gt;Hi Alex,&lt;/p&gt;

&lt;p&gt;Have you had a chance to upgrade your system already? If so, please let us know if the issue reoccurred or if 2.6 fixed the problem for you. &lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="577458" author="renctan" created="Thu, 8 May 2014 21:21:23 +0000"  >&lt;p&gt;I tried following the steps in 2.4.9 and I was able to see the socket exception from the mongos, but it does resolve by itself afterwards so I am not sure if we were seeing the same thing. It is still possible to see the same error in 2.6, but since we actively check the status of the connections, the occurrence of this issue is minimized.&lt;/p&gt;</comment>
                            <comment id="577442" author="apiggott@ikanow.com" created="Thu, 8 May 2014 21:12:14 +0000"  >&lt;p&gt;Randolph - thanks for the update, we haven&apos;t moved across to 2.6.1 yet, though it&apos;s on our TODO list (we&apos;re stockpiling empty chunks!), we&apos;ll do that in the next couple of weeks and report back 6 weeks later on whether the issues have stopped &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;

&lt;p&gt;Though seems like Jeremie had possibly found a simple way to reproduce so checking if that is fixed would be a good start&lt;/p&gt;</comment>
                            <comment id="577384" author="renctan" created="Thu, 8 May 2014 20:33:37 +0000"  >&lt;p&gt;Have you tried v2.6.1? It has a fix for &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-9041&quot; title=&quot;proactively detect broken connections detected by the network&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-9041&quot;&gt;&lt;del&gt;SERVER-9041&lt;/del&gt;&lt;/a&gt; that should alleviate most of the bad connection cleanup issues.&lt;/p&gt;</comment>
                            <comment id="562165" author="jcharest" created="Thu, 24 Apr 2014 20:43:05 +0000"  >&lt;p&gt;We are experiencing the same behavior on a test shard : 2 repsets (1 primary and 2 slaves on each). Restarting the mongos server &quot;fix&quot; it. The bug &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-9041&quot; title=&quot;proactively detect broken connections detected by the network&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-9041&quot;&gt;&lt;del&gt;SERVER-9041&lt;/del&gt;&lt;/a&gt; and comment the last comment seem interesting.&lt;/p&gt;

&lt;p&gt;Steps to reproduce :&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;stop the primary of one of the repset (4-dev-db-salt)&lt;/li&gt;
	&lt;li&gt;wait until a slave become primary
	&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
		&lt;li&gt;mongos connections are working at this point&lt;/li&gt;
		&lt;li&gt;rs.status() is fine&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
	&lt;li&gt;restart 4-dev-db-salt
	&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
		&lt;li&gt;a priority was assigned to it so it became primary&lt;/li&gt;
		&lt;li&gt;rs.status() is fine&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
	&lt;li&gt;after a few time, the error log will be print and the mongos will be unusable for this replica set&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;infos :&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;all mongos, mongod are on 2.4.8&lt;/li&gt;
	&lt;li&gt;CentOS 6.5&lt;/li&gt;
	&lt;li&gt;internal vm with near zero latency&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;mongos logs:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Thu Apr 24 15:50:58.631 [conn26] warning: socket exception when initializing on repset2:repset2/4-dev-db-salt:27017, 5-dev-db-salt:27018, 6-dev-db-salt:27018, current connection state is &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; { state: &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;		{ &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;		  conn: &quot;repset2/4-dev-db-salt:27017,5-dev-db-salt:27018,6-dev-db-salt:27018&quot;,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;		  vinfo: &quot;mmoTemplate.AvatarCollection @ 2|5||535803133cdb3ed5967912f4&quot;,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;		  cursor: &quot;(empty)&quot;, count: 0, done: false&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;		},&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;	retryNext: false,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;	init: false,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;	finish: false,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;	errored: false&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;:: caused by :: 9001 socket exception [SEND_ERROR] server [4-dev-db-salt:27017]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="523629" author="apiggott@ikanow.com" created="Wed, 26 Mar 2014 14:53:12 +0000"  >&lt;p&gt;Examples from log file when it occurs:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;mongos.log.2014-03-21T13-23-45-Fri Mar 21 10:00:27.238 [conn228999] Socket say send() errno:110 Connection timed out 10.45.142.86:27025&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;mongos.log.2014-03-21T13-23-45:Fri Mar 21 10:00:27.251 [conn228999] warning: socket exception when initializing on replica_set8:replica_set8/10.45.142.211:27025,10.45.142.86:27025, current connection state is { state: { conn: &quot;replica_set8/10.45.142.211:27025,10.45.142.86:27025&quot;, vinfo: &quot;doc_metadata.metadata @ 1869|3||528edf5f3a901fb6bb300289&quot;, cursor: &quot;(none)&quot;, count: 0, done: false }, retryNext: false, init: false, finish: false, errored: false } :: caused by :: 9001 socket exception [SEND_ERROR] server [10.45.142.86:27025]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;mongos.log.2014-03-21T13-23-45-Fri Mar 21 11:00:14.873 [conn228999] Socket say send() errno:110 Connection timed out 10.170.15.12:27020&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;mongos.log.2014-03-21T13-23-45-Fri Mar 21 11:00:14.874 [conn228999] Socket say send() errno:110 Connection timed out 10.33.131.5:27022&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;mongos.log.2014-03-21T13-23-45-Fri Mar 21 11:00:14.876 [conn228999] Socket say send() errno:110 Connection timed out 10.45.142.86:27025&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;mongos.log.2014-03-21T13-23-45:Fri Mar 21 11:00:14.876 [conn228999] warning: socket exception when initializing on replica_set8:replica_set8/10.45.142.211:27025,10.45.142.86:27025, current connection state is { state: { conn: &quot;replica_set8/10.45.142.211:27025,10.45.142.86:27025&quot;, vinfo: &quot;doc_metadata.metadata @ 1869|3||528edf5f3a901fb6bb300289&quot;, cursor: &quot;(none)&quot;, count: 0, done: false }, retryNext: false, init: false, finish: false, errored: false } :: caused by :: 9001 socket exception [SEND_ERROR] server [10.45.142.86:27025]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;Interesting, I hadn&apos;t noticed the timeouts when the problem occurred before the &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-9022&quot; title=&quot;Enable mongos mode which releases ShardConnections to the pool after read ops&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-9022&quot;&gt;&lt;del&gt;SERVER-9022&lt;/del&gt;&lt;/a&gt; fix. Fwiw there is very little chance the IPs listed are not responding.&lt;/p&gt;

&lt;p&gt;(EDIT: in case it&apos;s causing confusion, note the mongod instances live on port (27017 + shardid) where in this case shardid=1...10, mongos lives on 27017, and the configdb lives on port 27016)&lt;/p&gt;

&lt;p&gt;EDIT2: note the significance of the timestamps is that there&apos;s an hourly test that runs which calls ,amongst other things, db.collection.count() which seems to be the call that fails the most frequently. Note also it&apos;s the same (stale?) connection being used in both cases.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="50197">SERVER-7008</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>12.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 24 Apr 2014 20:43:05 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        9 years, 5 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/WRITING-1126'>WRITING-1126</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>jantje.daniel</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            9 years, 5 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10249"><![CDATA[Cannot]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>adamc</customfieldvalue>
            <customfieldvalue>apiggott@ikanow.com</customfieldvalue>
            <customfieldvalue>jcharest</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
            <customfieldvalue>randolph@mongodb.com</customfieldvalue>
            <customfieldvalue>sflint@sailthru.com</customfieldvalue>
            <customfieldvalue>srinivasa_kanamatha@mcafee.com</customfieldvalue>
            <customfieldvalue>thomas.rueckstiess@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrly3r:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrxg1z:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>109185</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11861" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>User Summary</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="11858"><![CDATA[Completed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrm9nj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>