<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:05:42 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-4340] Query not returning data from specific shard when querying on anything but &quot;_id&quot; only</title>
                <link>https://jira.mongodb.org/browse/SERVER-4340</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hi,&lt;/p&gt;

&lt;p&gt;We&apos;ve noticed that on one of our sharded collection we cannot do any non-specific queries. For example:&lt;/p&gt;

&lt;p&gt;db.log_service.find() returns nothing, querying on anything other than &lt;del&gt;only&lt;/del&gt; _id fails.&lt;br/&gt;
db.log_service.find(&lt;/p&gt;
{&quot;_id&quot;:ObjectId(&quot;4eb3bf877961dc244d008235&quot;)}
&lt;p&gt;) returns the document.&lt;/p&gt;

&lt;p&gt;The environment is sharded over 3 replicaton sets, all of which currently only have a primary and an arbiter. The secondary has been taken down due to issues non-related to this. The configuration of the replicasets has been adjusted and rs.reconfig() has been run on all primaries. We have noticed however that mongos keeps complaining in the logs about not being able to reach these servers. Not sure if this is related to the issue.&lt;/p&gt;

&lt;p&gt;Having previously discussed this on IRC, I&apos;ve prepared a few pastes:&lt;/p&gt;

&lt;p&gt;db.printShardingStatus();&lt;br/&gt;
&lt;a href=&quot;http://pastie.org/private/pufkfflhtkwifmgcq8aq&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://pastie.org/private/pufkfflhtkwifmgcq8aq&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;mongos log extract:&lt;br/&gt;
&lt;a href=&quot;http://pastie.org/private/93el01g9nocz9ocfiqzdw&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://pastie.org/private/93el01g9nocz9ocfiqzdw&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;rs.status() on replSetB primary (the only shard that has data for that collection as an automated script removed the data from the other shards, not from this one as aparently it couldn&apos;t find any):&lt;br/&gt;
&lt;a href=&quot;http://pastie.org/private/an8g0gun7l6kevffusmytw&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://pastie.org/private/an8g0gun7l6kevffusmytw&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;explain() on the working query:&lt;br/&gt;
  db.log_service.find(&lt;/p&gt;
{&quot;_id&quot;:ObjectId(&quot;4eb3bf877961dc244d008235&quot;)}
&lt;p&gt;)&lt;br/&gt;
  &lt;a href=&quot;http://pastie.org/private/edfgbh9vkhlwf7o8tjmdq&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://pastie.org/private/edfgbh9vkhlwf7o8tjmdq&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;explain() on the failing query:&lt;br/&gt;
  db.log_service.find()&lt;br/&gt;
  &lt;a href=&quot;http://pastie.org/private/igeth69f84agwdjhmycyqa&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://pastie.org/private/igeth69f84agwdjhmycyqa&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;We first noticed the issue when we couldn&apos;t query data with skip() set to a number higher than what was available on replSetA. At that time find() without limit and skip worked. Now, replSetA is empty and these stopped working.&lt;br/&gt;
Queries with skip() higher than item count on replSetA for &lt;del&gt;different&lt;/del&gt; collections works without issues.&lt;/p&gt;

&lt;p&gt;Best regards,&lt;br/&gt;
Koen&lt;/p&gt;</description>
                <environment>Debian Squeeze</environment>
        <key id="25217">SERVER-4340</key>
            <summary>Query not returning data from specific shard when querying on anything but &quot;_id&quot; only</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="5">Cannot Reproduce</resolution>
                                        <assignee username="spencer@mongodb.com">Spencer Brody</assignee>
                                    <reporter username="koen@layer7.be">Koen Calliauw</reporter>
                        <labels>
                            <label>limit</label>
                            <label>replicaset</label>
                            <label>sharding</label>
                            <label>skip</label>
                    </labels>
                <created>Mon, 21 Nov 2011 15:34:01 +0000</created>
                <updated>Fri, 8 Mar 2013 15:56:11 +0000</updated>
                            <resolved>Wed, 5 Sep 2012 17:24:59 +0000</resolved>
                                    <version>2.0.1</version>
                                                    <component>Querying</component>
                    <component>Replication</component>
                    <component>Sharding</component>
                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="134937" author="spencer" created="Wed, 20 Jun 2012 19:16:51 +0000"  >&lt;p&gt;Hi Koen,&lt;br/&gt;
Is this still happening for you?  I have not been able to reproduce this successfully...&lt;br/&gt;
Would it be possible to upload a dump of your data?  Or a smaller dataset that reproduces the problem?&lt;/p&gt;</comment>
                            <comment id="86013" author="koen@layer7.be" created="Tue, 7 Feb 2012 09:46:51 +0000"  >&lt;p&gt;Hi Spencer,&lt;/p&gt;

&lt;p&gt;This is still the case, however, something changed, please see the explain() below, which was ran from mongos. Notice how n: is now 0. We&apos;ve upgraded everything to 2.0.2 in the meantime.&lt;/p&gt;

&lt;p&gt;explain() from mongos: &lt;a href=&quot;http://pastie.org/private/pmwbd1ov1r7f8pzdjmucg&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://pastie.org/private/pmwbd1ov1r7f8pzdjmucg&lt;/a&gt;&lt;br/&gt;
explain() from mongo2: &lt;a href=&quot;http://pastie.org/private/p2nvw8qqhm8amaa7ftrnea&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://pastie.org/private/p2nvw8qqhm8amaa7ftrnea&lt;/a&gt; (primary of ReplSetB and the only shard with data in it)&lt;/p&gt;

&lt;p&gt;mongos log (LogLevel 10):&lt;br/&gt;
&lt;a href=&quot;http://pastie.org/private/nleq4fluaig5y9gdcl3uw&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://pastie.org/private/nleq4fluaig5y9gdcl3uw&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The mongod&apos;s I can&apos;t set the LogLevel to 10 (error: &lt;a href=&quot;http://pastie.org/private/lhzxcjw6g1jlq8sqknsra&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://pastie.org/private/lhzxcjw6g1jlq8sqknsra&lt;/a&gt;)&lt;br/&gt;
mongod1 log:&lt;br/&gt;
&lt;a href=&quot;http://pastie.org/private/zuf43urxsppl6uj9k0tqg&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://pastie.org/private/zuf43urxsppl6uj9k0tqg&lt;/a&gt;&lt;br/&gt;
mongod2 log:&lt;br/&gt;
&lt;a href=&quot;http://pastie.org/private/1l1iii1efwmb2iukba67cg&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://pastie.org/private/1l1iii1efwmb2iukba67cg&lt;/a&gt;&lt;br/&gt;
mongod3 log:&lt;br/&gt;
&lt;a href=&quot;http://pastie.org/private/hwzy630izsjy8otkvq3req&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://pastie.org/private/hwzy630izsjy8otkvq3req&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;So, to answer your question, this is still happening but we can only reproduce it on this collection with this specific data (hasn&apos;t happened anywhere else yet).&lt;br/&gt;
Hope this helps. &lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Koen&lt;/p&gt;</comment>
                            <comment id="81018" author="spencer" created="Thu, 19 Jan 2012 17:51:22 +0000"  >&lt;p&gt;Hi Koen.  Is this still happening for you?  Have you tried upgrading to 2.0.2?&lt;/p&gt;</comment>
                            <comment id="74449" author="spencer" created="Mon, 19 Dec 2011 17:07:45 +0000"  >&lt;p&gt;This is the same paste you attached earlier in the ticket using log level of 5.  Can you capture a new log of trying the query and having it fail while running with loglevel 10?  I&apos;d also like logs form the primary of every shard.  You can attach files to this ticket, and if you zip/tar the log files you should be able to just attach the full files directly.&lt;/p&gt;</comment>
                            <comment id="73777" author="koen@layer7.be" created="Fri, 16 Dec 2011 03:11:26 +0000"  >&lt;p&gt;Hi Spencer,&lt;/p&gt;

&lt;p&gt;You can find the paste here:&lt;br/&gt;
&lt;a href=&quot;http://pastie.org/private/p1frdlftdmbxxuqqq18rbg&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://pastie.org/private/p1frdlftdmbxxuqqq18rbg&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The actual query happened at 09:58:53&lt;/p&gt;

&lt;p&gt;Best regards,&lt;br/&gt;
Koen Calliauw&lt;br/&gt;
Layer7 v.o.f.&lt;br/&gt;
DDI: +32 50 704 415&lt;br/&gt;
Cell: +32 499 74 28 73&lt;/p&gt;




</comment>
                            <comment id="72086" author="spencer" created="Fri, 9 Dec 2011 20:01:47 +0000"  >&lt;p&gt;Actually, for good measure, please attach the logs from the primaries of each shard for the same period as well.  To confirm, you&apos;re not doing slaveOk queries, right?&lt;/p&gt;</comment>
                            <comment id="72085" author="spencer" created="Fri, 9 Dec 2011 20:01:02 +0000"  >&lt;p&gt;I want the logs from the mongos running 2.0.1 at log level 10 for when you try to do a generic query and it fails.&lt;/p&gt;</comment>
                            <comment id="71923" author="koen@layer7.be" created="Fri, 9 Dec 2011 10:23:43 +0000"  >&lt;p&gt;Hi Spencer,&lt;/p&gt;

&lt;p&gt;Can you tell me from which timeframe you need these logs and from which servers? Also, on 1.8.4 or 2.0.1?&lt;/p&gt;

&lt;p&gt;/Koen&lt;/p&gt;</comment>
                            <comment id="71437" author="spencer" created="Wed, 7 Dec 2011 17:17:56 +0000"  >&lt;p&gt;Can you attach the logs from log level 10?  There&apos;s one specific message I&apos;m looking for.  As for the &quot;shard version not ok&quot; error, that&apos;s from &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-4020&quot; title=&quot;bad serverID set in setShardVersion&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-4020&quot;&gt;&lt;del&gt;SERVER-4020&lt;/del&gt;&lt;/a&gt; which is fixed in 2.0.2 - you can either try upgrading your mongos to the 2.0.2 release candidate or wait for the official 2.0.2 release that should be happening in about a week.  As you&apos;ve already figured out, the workaround is to run flushRouterConfig whenever it happens.&lt;/p&gt;</comment>
                            <comment id="71363" author="koen@layer7.be" created="Wed, 7 Dec 2011 09:19:23 +0000"  >&lt;p&gt;Believe it or not, but the query works fine with 1.8.4 mongos! (waiting about 1 minute after mongos was started. Any earlier again didn&apos;t return anything)&lt;/p&gt;

&lt;p&gt;after reverting back to 2.0.1, query stopped working again.&lt;/p&gt;

&lt;p&gt;I actually don&apos;t see any additional stuff in the logs when setting the loglevel to 10. I ran the command through mongos. &lt;/p&gt;

&lt;p&gt;What I also noticed the last few days is that I get these messages:&lt;br/&gt;
  Wed Dec  7 09:06:08 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn176&amp;#93;&lt;/span&gt; Assertion: 13388:&lt;span class=&quot;error&quot;&gt;&amp;#91;betrobot_production.bets&amp;#93;&lt;/span&gt; shard version not ok in Client::Context: client in sharded mode, but doesn&apos;t have version set for this collection: betrobot_production.bets myVersion: 9|0&lt;br/&gt;
on a specific primary (only one at a time usually) at a &lt;em&gt;very&lt;/em&gt; fast rate (also filling up disk). Running the flushRouterConfig fixes that, but having to run that every time on all mongos is a pain in the backside to say the least &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/wink.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt; &lt;/p&gt;

&lt;p&gt;/Koen&lt;/p&gt;</comment>
                            <comment id="71020" author="spencer" created="Mon, 5 Dec 2011 23:49:24 +0000"  >&lt;p&gt;Also, could you try it one more time on 2.0.1 after setting the log level to 10 (sorry for making you do that twice, I thought 5 was the highest log level but actually realized that there&apos;s some useful information at levels higher than that).  After getting the log output with level 10 verbosity you can set the log level back to normal to avoid filling up your disk too quickly.&lt;/p&gt;</comment>
                            <comment id="71018" author="spencer" created="Mon, 5 Dec 2011 23:45:36 +0000"  >&lt;p&gt;Could you try bringing up a 1.8.4 mongos and see if you see the same problem when querying through it or if any different errors are thrown when using 1.8.4?  It should be fine to use mongos from 1.8.4 with a 2.0.1 cluster.&lt;/p&gt;</comment>
                            <comment id="70828" author="koen@layer7.be" created="Mon, 5 Dec 2011 11:40:51 +0000"  >&lt;p&gt;Hi, &lt;/p&gt;

&lt;p&gt;Anything else I can try? Data is still stuck on this server. &lt;/p&gt;

&lt;p&gt;/Koen&lt;/p&gt;

&lt;p&gt;PS: I also reset the loglevel back now. Disk filled up after a few hours.&lt;/p&gt;</comment>
                            <comment id="69496" author="koen@layer7.be" created="Tue, 29 Nov 2011 10:38:51 +0000"  >&lt;p&gt;Hi Spencer,&lt;/p&gt;

&lt;p&gt;You can find the paste here:&lt;br/&gt;
&lt;a href=&quot;http://pastie.org/private/p1frdlftdmbxxuqqq18rbg&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://pastie.org/private/p1frdlftdmbxxuqqq18rbg&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;/Koen&lt;/p&gt;</comment>
                            <comment id="69325" author="spencer" created="Mon, 28 Nov 2011 18:49:52 +0000"  >&lt;p&gt;Can you try upping the logging verbosity on the mongos by running&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;db.adminCommand( { setParameter : 1 , logLevel : 5 } )&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;And then try querying and send the log output?&lt;/p&gt;</comment>
                            <comment id="69167" author="koen@layer7.be" created="Mon, 28 Nov 2011 07:33:41 +0000"  >&lt;p&gt;Yes, everything has even been rebooted in the meantime, so all mongod, mongos and config servers have been restarted at least once or twice since the issue started showing.&lt;/p&gt;

&lt;p&gt;/Koen&lt;/p&gt;</comment>
                            <comment id="69160" author="eliot" created="Mon, 28 Nov 2011 06:23:47 +0000"  >&lt;p&gt;Have you tried bouncing all mongod and mongos?&lt;/p&gt;</comment>
                            <comment id="69050" author="koen@layer7.be" created="Sun, 27 Nov 2011 07:27:09 +0000"  >&lt;p&gt;Also, I doubt that removing the secondaries has anything to do with this issue, since we were able to query the 2 other shards perfectly, also after removing the replset secondaries there.&lt;/p&gt;

&lt;p&gt;/Koen&lt;/p&gt;</comment>
                            <comment id="69049" author="koen@layer7.be" created="Sun, 27 Nov 2011 07:24:19 +0000"  >&lt;p&gt;Hi Eliot,&lt;/p&gt;

&lt;p&gt;Right now, we don&apos;t have an active secondary, no. Due to technical issues we had to take them offline temporary. We won&apos;t be able to add them back in until this issue is fixed.&lt;/p&gt;

&lt;p&gt;We don&apos;t know precisely when it started happening. We had a script that emptied these collections and puts it on another server. The issue was only visible when the script had finished and we noticed that there was still data left on replSetB which we could not query in the usual way. The replicaset secondaries were removed somewhere at the same time.&lt;/p&gt;

&lt;p&gt;Repairs happened on several occasions to make some diskspace.&lt;/p&gt;

&lt;p&gt;/Koen&lt;/p&gt;</comment>
                            <comment id="69040" author="eliot" created="Sun, 27 Nov 2011 04:21:45 +0000"  >&lt;p&gt;You have a secondary for each shard though, right?&lt;/p&gt;

&lt;p&gt;When did this error start happening?&lt;/p&gt;

&lt;p&gt;Was there ever a repair()?&lt;/p&gt;

&lt;p&gt;Sorry for all the questions - just hard to understand exactly what&apos;s going on.&lt;/p&gt;</comment>
                            <comment id="68993" author="koen@layer7.be" created="Sat, 26 Nov 2011 17:36:46 +0000"  >&lt;p&gt;Hi Eliot,&lt;/p&gt;

&lt;p&gt;The bets and log_service were sharded at the same time if I remember correctly. Every shard is a replica set. As mentioned previously every replica set only has a primary and an arbiter for the time being. This isn&apos;t an issue I suppose? We only implemented the replicasets for redundancy, for now, we never queried with the option to prefer slaves.&lt;/p&gt;

&lt;p&gt;Besides the technical issues (diskspace) we had no previous issues with the replicasets. &lt;br/&gt;
The only issue we had previously with this setup was a bug in Shanty (a mongodb adaptor for Zend Framework). We&apos;ve worked around this issue by adapting our code (not using the export() function of Shanty). The issue here was that cursors didn&apos;t get closed correctly and the mongos processes grew bigger and bigger in resident memory size during times of high load.&lt;/p&gt;

&lt;p&gt;/Koen&lt;/p&gt;</comment>
                            <comment id="68981" author="eliot" created="Sat, 26 Nov 2011 14:38:32 +0000"  >&lt;p&gt;When did you shard bets and log_service?&lt;/p&gt;

&lt;p&gt;What about the replica sets?&lt;/p&gt;

&lt;p&gt;Any issues on those recently?&lt;/p&gt;</comment>
                            <comment id="68980" author="koen@layer7.be" created="Sat, 26 Nov 2011 14:32:24 +0000"  >&lt;p&gt;Hi,&lt;/p&gt;

&lt;p&gt;The db.stats() printout of the primaries of the 3 shards can be found here:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;http://pastie.org/private/ketwnaca5ufrar3mwlfq&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://pastie.org/private/ketwnaca5ufrar3mwlfq&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;A bit of history, well, we&apos;ve been running this sharded setup for about 3 months in production now, give or take a few days, but due to disk space constraints we had to turn logappend off, so unfortunately I don&apos;t have logs going back further than yesterday &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/sad.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;br/&gt;
Before the sharded production setup we ran the same code (php) on a single mongodb node for another 2 months (estimate again). So basically sharding has been added on the collections from day 1 in this setup. Before that, it was on a totally different server.&lt;br/&gt;
Basically every server fulfills 4 roles:&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;primary of shard x&lt;/li&gt;
	&lt;li&gt;secondary of shard y&lt;/li&gt;
	&lt;li&gt;arbiter of shard z&lt;/li&gt;
	&lt;li&gt;config server&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;We chose this type of config on this setup because we are quite restrained on budget and resources. This is running on a hosted environment based on KVM virtualization without overcommitment of cpu or memory. &lt;/p&gt;

&lt;p&gt;If you need more history, can you please be a bit more specific on what you mean by that? I&apos;ll gladly provide you with any and all information I can give you. &lt;/p&gt;

&lt;p&gt;/Koen&lt;/p&gt;</comment>
                            <comment id="68979" author="eliot" created="Sat, 26 Nov 2011 14:12:44 +0000"  >&lt;p&gt;Can you run db.stats() on each of the shards?&lt;/p&gt;

&lt;p&gt;Can you explain a bit more of the history.&lt;/p&gt;

&lt;p&gt;When did you add sharding for example.&lt;/p&gt;

&lt;p&gt;Do you have the logs going back that far?&lt;/p&gt;</comment>
                            <comment id="68968" author="koen@layer7.be" created="Sat, 26 Nov 2011 08:42:01 +0000"  >&lt;p&gt;Yesterday we rebooted the app servers and all mongo servers for some hardware changes. All stayed the same afterward. Still cannot query data from that collection.&lt;/p&gt;</comment>
                            <comment id="68500" author="koen@layer7.be" created="Wed, 23 Nov 2011 08:38:33 +0000"  >&lt;p&gt;mongod logs from replSet primaries&lt;br/&gt;
mongos log&lt;/p&gt;</comment>
                            <comment id="68499" author="koen@layer7.be" created="Wed, 23 Nov 2011 08:37:25 +0000"  >&lt;p&gt;From the mailinglist:&lt;/p&gt;

&lt;p&gt;Everything (all mongos and mongod&apos;s) is running 2.0.1.&lt;br/&gt;
Here are the explain()s for the failing query against the shards, but please note that only the second shard (replSetB) has data in it. &lt;/p&gt;

&lt;p&gt;Primary of replSetA: &lt;a href=&quot;http://pastie.org/2907612&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://pastie.org/2907612&lt;/a&gt;&lt;br/&gt;
Primary of replSetB: &lt;a href=&quot;http://pastie.org/2907616&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://pastie.org/2907616&lt;/a&gt; &lt;br/&gt;
Primary of replSetC: &lt;a href=&quot;http://pastie.org/2907620&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://pastie.org/2907620&lt;/a&gt;&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;I don&apos;t know why the chunks are still there on replSetA and replSetC (as seen in printShardingStatus()). I assumed this was normal and that on deletes these chunks stayed there as preallocation for future data. Also, when I mentioned replSetA (and replSetC) were empty, I meant that there are no documents in the collection in question (log_service).&lt;/li&gt;
	&lt;li&gt;I&apos;ll attach the requested logs to the issue.&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="68294" author="spencer" created="Tue, 22 Nov 2011 18:54:30 +0000"  >&lt;p&gt;You said that replSetB is the shard with the data, and you also mentioned that replSetA is empty.  What do you mean by that?  The output of printShardingStatus() shows many chunks living on replSetA.&lt;/p&gt;

&lt;p&gt;Can you also attach the full mongos log as well as the full logs from the primaries of all 3 replSets?  Log files generally compress very well, so if you zip/tar the files they should be quite small and can be attached to this ticket no problem.&lt;/p&gt;</comment>
                            <comment id="68176" author="koen@layer7.be" created="Tue, 22 Nov 2011 09:08:07 +0000"  >&lt;p&gt;In the mongod log if the primary in shard replSetB (the one with the data) we see the queries:&lt;br/&gt;
    Tue Nov 22 09:03:01 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn6&amp;#93;&lt;/span&gt; query betrobot_production.log_service nscanned:138718 reslen:20 293ms&lt;/p&gt;

&lt;p&gt;I&apos;ve also noticed this warning. Don&apos;t know if it&apos;s important:&lt;br/&gt;
    Tue Nov 22 09:03:18 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn110&amp;#93;&lt;/span&gt; warning: bad serverID set in setShardVersion and none in info: EOO&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="13920" name="mongologs.tar.gz" size="3914087" author="koen@layer7.be" created="Wed, 23 Nov 2011 08:38:33 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>29.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 22 Nov 2011 18:54:30 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        11 years, 35 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ian@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            11 years, 35 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10020"><![CDATA[Linux]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>eliot</customfieldvalue>
            <customfieldvalue>koen@layer7.be</customfieldvalue>
            <customfieldvalue>spencer@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrokzb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrg4kf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7644</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrli4f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>