<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:14:08 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-7297] Queries against sharded collection return &quot;all servers down&quot;</title>
                <link>https://jira.mongodb.org/browse/SERVER-7297</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Two replication sets, each with 3 servers. Three mongod config servers. Two mongos application servers. I created a sharded configuration with the two replication sets and sharded some collections. After bouncing all the servers, queries in the shell against non-sharded collections succeed, but queries against sharded collections return:&lt;/p&gt;

&lt;p&gt;MongoDB shell version: 2.2.0&lt;br/&gt;
connecting to: mongos:27017/test&lt;br/&gt;
mongos&amp;gt; use policydb;                                                                                                     &lt;br/&gt;
switched to db policydb&lt;br/&gt;
mongos&amp;gt; db.policies.findOne();                                                                                            &lt;br/&gt;
Mon Oct  8 21:44:44 uncaught exception: error {&lt;br/&gt;
        &quot;$err&quot; : &quot;setShardVersion failed host: cw-mongodb2-test:27017 &lt;/p&gt;
{ oldVersion: Timestamp 0|0, oldVersionEpoch: ObjectId(&apos;000000000000000000000000&apos;), errmsg: \&quot;exception: all servers down!\&quot;, code: 8002, ok: 0.0 }
&lt;p&gt;&quot;,&lt;br/&gt;
        &quot;code&quot; : 10429&lt;br/&gt;
}&lt;/p&gt;</description>
                <environment>Ubuntu Linux 12.04. All servers are at the 2.2.0 version.</environment>
        <key id="52629">SERVER-7297</key>
            <summary>Queries against sharded collection return &quot;all servers down&quot;</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="spencer@mongodb.com">Spencer Brody</assignee>
                                    <reporter username="smcqueen">Stan McQueen</reporter>
                        <labels>
                    </labels>
                <created>Mon, 8 Oct 2012 21:45:33 +0000</created>
                <updated>Fri, 15 Feb 2013 15:06:32 +0000</updated>
                            <resolved>Wed, 2 Jan 2013 23:31:37 +0000</resolved>
                                    <version>2.2.0</version>
                                                    <component>Sharding</component>
                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="223285" author="saltycookie@gmail.com" created="Mon, 24 Dec 2012 11:16:09 +0000"  >&lt;p&gt;I fixed it follow your hints. Thanks a lot!&lt;/p&gt;</comment>
                            <comment id="222915" author="smcqueen" created="Sun, 23 Dec 2012 20:30:08 +0000"  >&lt;p&gt;My issue turned out to be a missing host file entry. Each server was accessible from every other server, but not under the host names that the replication set was created with. Check your mongo logs carefully to see if there is a &quot;failure to connect&quot; message in one or more of them. That is what eventually led to my finding the resolution of the problem.&lt;/p&gt;</comment>
                            <comment id="222676" author="saltycookie@gmail.com" created="Sun, 23 Dec 2012 10:08:57 +0000"  >&lt;p&gt;I am encounting such a problem, too. My mongo cluster&apos;s setting are almost the same as Stan&apos;s. The connections between mongos, config servers and replica sets have been checked, and there seems to have no problem.&lt;/p&gt;</comment>
                            <comment id="181591" author="smcqueen" created="Thu, 1 Nov 2012 20:53:19 +0000"  >&lt;p&gt;While I was preparing the logs to include, I did notice a &quot;failure to connect&quot; message. Upon looking into it, I found that I had omitted a host entry for the mongod servers. Interestingly, addShard and enableSharding worked and did not cause a problem with queries; but when shardCollection was executed, the command succeeded and sh.status seemed to indicate that all was well even though queries would fail. Now that I have added the missing host entries, all is working as expected. Thanks for your help and I apologize for entering a bug that was actually all my own fault.&lt;/p&gt;</comment>
                            <comment id="177316" author="spencer" created="Fri, 19 Oct 2012 20:00:03 +0000"  >&lt;p&gt;Can you include a larger section of the log, with at least 10 minutes to either side of the error?&lt;/p&gt;

&lt;p&gt;This looks a lot like a problem with the connection between the shard mongods and the config servers.  Can you double-check that it is possible to connect to the config servers via the mongo shell from the shard mongod servers?&lt;/p&gt;</comment>
                            <comment id="177308" author="smcqueen" created="Fri, 19 Oct 2012 19:33:29 +0000"  >&lt;p&gt;The problem still exists if I enable sharding on a collection. Once I do that, no further queries on that collection succeed. The only remedy is to delete the config server databases and the local databases on the replica set members, restart everything, re-establish the replica set, and create the shard set again. As long as I don&apos;t actually enable sharding for a collection, the mongoses, mongods, and mongodbs respond normally. All servers are reachable from all other servers. Currently I am running with a single shard of one replica set consisting of three servers, three config dbs, and two mongoses. Since restarting everything, I have created the shard, adding the replica set to it, but have not enabled sharding for any of the databases or collections. In that mode, everything works fine. However, this is a load-testing cluster, and I would like to be able to test the effects of sharding on the ability to handle load (before our production servers actually get to the point of having high loads).&lt;/p&gt;

&lt;p&gt;Our production servers are running MongoDB version 2.0.2 with a single shard of a replica set of three servers. Sharding is enabled on several databases/collections and these servers do not exhibit these symptoms.&lt;/p&gt;

&lt;p&gt;My first assumption was that I had somehow screwed up the configuration, since I couldn&apos;t believe a major sharding bug could have gotten out into the wild, but my load-test config is virtually identical to the production config.&lt;/p&gt;</comment>
                            <comment id="177204" author="spencer" created="Fri, 19 Oct 2012 17:28:15 +0000"  >&lt;p&gt;Hi Stan,&lt;br/&gt;
Sorry for the delayed response.  Are you still experiencing this problem?  If so, have you tried bouncing your mongoses?  Can you confirm that you can create a connection to each of your config servers from the machine that is running your mongos as well as from each of the machines in your replica set?&lt;/p&gt;</comment>
                            <comment id="173137" author="smcqueen" created="Tue, 9 Oct 2012 22:24:04 +0000"  >&lt;p&gt;By the way, the same symptoms occur with a sharded environment consisting of a single replication set.&lt;/p&gt;</comment>
                            <comment id="173120" author="smcqueen" created="Tue, 9 Oct 2012 21:49:56 +0000"  >&lt;p&gt;Here is the query and response:&lt;br/&gt;
mongos&amp;gt; use licensedb;&lt;br/&gt;
switched to db licensedb&lt;br/&gt;
mongos&amp;gt; db.licenses.findOne();&lt;br/&gt;
Tue Oct  9 21:46:00 uncaught exception: error {&lt;br/&gt;
	&quot;$err&quot; : &quot;setShardVersion failed host: cw-mongodb1-test:27017 &lt;/p&gt;
{ oldVersion: Timestamp 0|0, oldVersionEpoch: ObjectId(&apos;000000000000000000000000&apos;), errmsg: \&quot;exception: all servers down!\&quot;, code: 8002, ok: 0.0 }
&lt;p&gt;&quot;,&lt;br/&gt;
	&quot;code&quot; : 10429&lt;br/&gt;
}&lt;br/&gt;
==============================================&lt;br/&gt;
Here is the output of sh.status():&lt;br/&gt;
mongos&amp;gt; sh.status();&lt;br/&gt;
&amp;#8212; Sharding Status &amp;#8212; &lt;br/&gt;
  sharding version: &lt;/p&gt;
{ &quot;_id&quot; : 1, &quot;version&quot; : 3 }
&lt;p&gt;  shards:&lt;/p&gt;
	{  &quot;_id&quot; : &quot;cwset&quot;,  &quot;host&quot; : &quot;cwset/cw-mongodb1-test:27017,cw-mongodb2-test:27017,cw-mongodb3-test:27017&quot; }
	{  &quot;_id&quot; : &quot;cwset-2&quot;,  &quot;host&quot; : &quot;cwset-2/cw-mongodb1-2-test:27017,cw-mongodb2-2-test:27017,cw-mongodb3-2-test:27017&quot; }
&lt;p&gt;  databases:&lt;/p&gt;
	{  &quot;_id&quot; : &quot;admin&quot;,  &quot;partitioned&quot; : false,  &quot;primary&quot; : &quot;config&quot; }
	{  &quot;_id&quot; : &quot;licensedb&quot;,  &quot;partitioned&quot; : true,  &quot;primary&quot; : &quot;cwset&quot; }
&lt;p&gt;		licensedb.licenses chunks:&lt;br/&gt;
				cwset	1&lt;br/&gt;
			{ &quot;u&quot; : &lt;/p&gt;
{ $minKey : 1 }
&lt;p&gt; } --&amp;gt;&amp;gt; { &quot;u&quot; : &lt;/p&gt;
{ $maxKey : 1 }
&lt;p&gt; } on : cwset Timestamp(1000, 0) &lt;br/&gt;
		licensedb.seats chunks:&lt;br/&gt;
				cwset	1&lt;br/&gt;
			{ &quot;_machineHash&quot; : &lt;/p&gt;
{ $minKey : 1 }
&lt;p&gt; } --&amp;gt;&amp;gt; { &quot;_machineHash&quot; : &lt;/p&gt;
{ $maxKey : 1 }
&lt;p&gt; } on : cwset Timestamp(1000, 0) &lt;/p&gt;
	{  &quot;_id&quot; : &quot;oauthdb&quot;,  &quot;partitioned&quot; : false,  &quot;primary&quot; : &quot;cwset&quot; }
	{  &quot;_id&quot; : &quot;imagedb&quot;,  &quot;partitioned&quot; : false,  &quot;primary&quot; : &quot;cwset&quot; }
	{  &quot;_id&quot; : &quot;certdb&quot;,  &quot;partitioned&quot; : false,  &quot;primary&quot; : &quot;cwset&quot; }
	{  &quot;_id&quot; : &quot;policydb&quot;,  &quot;partitioned&quot; : true,  &quot;primary&quot; : &quot;cwset&quot; }
&lt;p&gt;		policydb.policies chunks:&lt;br/&gt;
				cwset	1&lt;br/&gt;
			{ &quot;u&quot; : &lt;/p&gt;
{ $minKey : 1 }
&lt;p&gt; } --&amp;gt;&amp;gt; { &quot;u&quot; : &lt;/p&gt;
{ $maxKey : 1 }
&lt;p&gt; } on : cwset Timestamp(1000, 0) &lt;/p&gt;
	{  &quot;_id&quot; : &quot;settingsdb&quot;,  &quot;partitioned&quot; : false,  &quot;primary&quot; : &quot;cwset&quot; }
	{  &quot;_id&quot; : &quot;statistics&quot;,  &quot;partitioned&quot; : false,  &quot;primary&quot; : &quot;cwset&quot; }
	{  &quot;_id&quot; : &quot;licenseb&quot;,  &quot;partitioned&quot; : false,  &quot;primary&quot; : &quot;cwset-2&quot; }

&lt;p&gt;=======================================================================&lt;br/&gt;
Here is the log from mongos starting before the query and ending after the query:&lt;br/&gt;
/usr/local/sbin/mongos(_ZN5mongo27ParallelSortClusteredCursor9startInitEv+0xe11) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x763581&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/local/sbin/mongos(_ZN5mongo27ParallelSortClusteredCursor8fullInitEv+0x9) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x769979&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/local/sbin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0x472) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x62a3f2&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/local/sbin/mongos(_ZN5mongo7Request7processEi+0x1fb) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x5c346b&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/local/sbin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x71) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x5003f1&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/local/sbin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x411) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x6b3731&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f8936d04e9a&amp;#93;&lt;/span&gt;&lt;br/&gt;
Tue Oct  9 21:46:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn1879&amp;#93;&lt;/span&gt; warning: db exception when initializing on cwset:cwset/cw-mongodb1-test:27017,cw-mongodb2-test:27017,cw-mongodb3-test:27017, current connection state is { state: &lt;/p&gt;
{ conn: &quot;cwset/cw-mongodb1-test:27017,cw-mongodb2-test:27017,cw-mongodb3-test:27017&quot;, vinfo: &quot;licensedb.licenses @ 1|0||507499c3e2c7beb86f78c174&quot;, cursor: &quot;(none)&quot;, count: 0, done: false }
&lt;p&gt;, retryNext: false, init: false, finish: false, errored: false } :: caused by :: 10429 setShardVersion failed host: cw-mongodb1-test:27017 &lt;/p&gt;
{ oldVersion: Timestamp 0|0, oldVersionEpoch: ObjectId(&apos;000000000000000000000000&apos;), errmsg: &quot;exception: all servers down!&quot;, code: 8002, ok: 0.0 }
&lt;p&gt;Tue Oct  9 21:46:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn1879&amp;#93;&lt;/span&gt; AssertionException while processing op type : 2004 to : licensedb.licenses :: caused by :: 10429 setShardVersion failed host: cw-mongodb1-test:27017 &lt;/p&gt;
{ oldVersion: Timestamp 0|0, oldVersionEpoch: ObjectId(&apos;000000000000000000000000&apos;), errmsg: &quot;exception: all servers down!&quot;, code: 8002, ok: 0.0 }
&lt;p&gt;Tue Oct  9 21:46:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn1879&amp;#93;&lt;/span&gt; Request::process begin ns: admin.$cmd msg id: 59 op: 2004 attempt: 0&lt;br/&gt;
Tue Oct  9 21:46:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn1879&amp;#93;&lt;/span&gt; single query: admin.$cmd  &lt;/p&gt;
{ replSetGetStatus: 1.0, forShell: 1.0 }
&lt;p&gt;  ntoreturn: -1 options : 0&lt;br/&gt;
Tue Oct  9 21:46:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn1879&amp;#93;&lt;/span&gt; Request::process end ns: admin.$cmd msg id: 59 op: 2004 &lt;/p&gt;</comment>
                            <comment id="172978" author="spencer" created="Tue, 9 Oct 2012 17:19:04 +0000"  >&lt;p&gt;Can you attach the output of running sh.status() in the shell?&lt;/p&gt;

&lt;p&gt;Can you also attach logs from the mongos from when you try to run a query on a sharded collection and see this error message instead?&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>10.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 9 Oct 2012 17:19:04 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        11 years, 8 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ian@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            11 years, 8 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>spencer@mongodb.com</customfieldvalue>
            <customfieldvalue>smcqueen</customfieldvalue>
            <customfieldvalue>saltycookie@gmail.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrnlun:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrjvsv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>29686</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht06d3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>