<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:16:46 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-8178] Odd Count / Document Results Differential On Sharded Collection</title>
                <link>https://jira.mongodb.org/browse/SERVER-8178</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;I get a count of 4 and a result of 2 documents when I do the following on a sharded collection&lt;/p&gt;

&lt;p&gt;mongos&amp;gt; db.agg_collection.find(&lt;/p&gt;
{&quot;_id.summary_id&quot;: &quot;c48b9bb63&quot;, &quot;_id.placement_id&quot;: &quot;1&quot;}
&lt;p&gt;).count()&lt;br/&gt;
4&lt;br/&gt;
mongos&amp;gt; db.agg_collection.find(&lt;/p&gt;
{&quot;_id.summary_id&quot;: &quot;c48b9bb63&quot;, &quot;_id.placement_id&quot;: &quot;1&quot;}
&lt;p&gt;).pretty()&lt;br/&gt;
{&lt;br/&gt;
	&quot;_id&quot; : &lt;/p&gt;
{
		&quot;placement_id&quot; : &quot;1&quot;,
		&quot;summary_id&quot; : &quot;c48b9bb63&quot;,
                &quot;new_or_used&quot; : &quot;N&quot;
	}
&lt;p&gt;,&lt;br/&gt;
	&quot;value&quot; : &apos;hidden for patent reasons&apos;&lt;br/&gt;
}&lt;br/&gt;
{&lt;br/&gt;
	&quot;_id&quot; : &lt;/p&gt;
{
		&quot;placement_id&quot; : &quot;1&quot;,
		&quot;summary_id&quot; : &quot;c48b9bb63&quot;,
                &quot;new_or_used&quot; : &quot;U&quot;
	}
&lt;p&gt;,&lt;br/&gt;
	&quot;value&quot; : &apos;hidden for patent reasons&apos;&lt;br/&gt;
}&lt;/p&gt;</description>
                <environment></environment>
        <key id="62027">SERVER-8178</key>
            <summary>Odd Count / Document Results Differential On Sharded Collection</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="aaron">Aaron Staple</assignee>
                                    <reporter username="wwatson">William Watson</reporter>
                        <labels>
                    </labels>
                <created>Tue, 15 Jan 2013 17:41:45 +0000</created>
                <updated>Tue, 26 Sep 2017 20:28:06 +0000</updated>
                            <resolved>Mon, 21 Jan 2013 03:13:10 +0000</resolved>
                                    <version>2.0.0</version>
                    <version>2.2.0</version>
                                                                        <votes>0</votes>
                                    <watches>6</watches>
                                                                                                                <comments>
                            <comment id="253995" author="wwatson" created="Wed, 30 Jan 2013 18:45:49 +0000"  >&lt;p&gt;Yeah, that probably would explain it. We WERE running on a sharded cluster with slave_ok on, but we are now running everything on a single mongoD until we are confident everything is right again with sharding, but I just wanted to make sure the community gets the information they need.&lt;/p&gt;</comment>
                            <comment id="253978" author="aaron" created="Wed, 30 Jan 2013 18:29:40 +0000"  >&lt;p&gt;Hi William - Are you just seeing that behavior when you allow slave ok queries?  If so, then &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-5931&quot; title=&quot;Secondary reads in sharded clusters need stronger consistency&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-5931&quot;&gt;&lt;del&gt;SERVER-5931&lt;/del&gt;&lt;/a&gt; is likely the cause.  I can update the &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-5665&quot; title=&quot;count command does not check chunk ranges&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-5665&quot;&gt;&lt;del&gt;SERVER-5665&lt;/del&gt;&lt;/a&gt; description to include the fact that it just describes the behavior on primaries, and &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-5931&quot; title=&quot;Secondary reads in sharded clusters need stronger consistency&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-5931&quot;&gt;&lt;del&gt;SERVER-5931&lt;/del&gt;&lt;/a&gt; represents addressing behavior on secondaries.  If you do not have slave ok queries enabled, then we can help diagnose the situation if you can provide a precise description of what differentiates the application layer queries from the non application layer queries.&lt;/p&gt;</comment>
                            <comment id="253796" author="wwatson" created="Wed, 30 Jan 2013 15:03:15 +0000"  >&lt;p&gt;Hi Aaron,&lt;/p&gt;

&lt;p&gt;I see your discussion on there. The phrase of note: &lt;/p&gt;

&lt;p&gt;&quot;When a query (not count) runs on a sharded cluster, each shard checks the documents that match the query to see if they also belong to chunks owned by that shard. If they do not belong to owned chunks, they are not returned.&quot; &lt;/p&gt;

&lt;p&gt;This is not correct. We are seeing exactly that situation, that documents which match the query and do not belong to the shard are returned, but only by the application-layer Ruby MongoDB driver.&lt;/p&gt;</comment>
                            <comment id="253538" author="aaron" created="Wed, 30 Jan 2013 07:51:31 +0000"  >&lt;p&gt;Hi William - A plan for properly handling this case is being discussed internally.  You can track our progress on implementing at &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-5665&quot; title=&quot;count command does not check chunk ranges&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-5665&quot;&gt;&lt;del&gt;SERVER-5665&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="252819" author="wwatson" created="Tue, 29 Jan 2013 14:47:51 +0000"  >&lt;p&gt;The aggregation framework won&apos;t work for us because what we&apos;re doing is pretty complex. However, I want to make myself clear about something that I don&apos;t think was really clear before.&lt;/p&gt;

&lt;p&gt;Your database, through the mongoDB ruby connector, is returning incorrect data because of these orphan documents. This was a HUGE issue for us and I&apos;m sure it is a huge issue for other companies as well. A database should not return incorrect data. Given that, I think your fixes for orphan documents should be patched into 2.2 as soon as humanly possible. Furthermore, I believe the ruby connector needs to be fixed since it was the connector returning  the wrong documents. The command line returned the wrong counts which is also an issue.&lt;/p&gt;</comment>
                            <comment id="245248" author="tyler@10gen.com" created="Mon, 21 Jan 2013 03:12:11 +0000"  >&lt;p&gt;Ok,&lt;/p&gt;

&lt;p&gt;Regarding slow map reduce, you can use the aggregation framework (2.2+) which avoids using the javascript engine (slow), allows for greater concurrency, and whose operations are expressed via a pipeline of json documents (easier to work with). Alternatively you may be very interested in Mongo 2.4 when released as we have switched the javascript engine to v8 from spider monkey (which also allows for greater concurrency). However, for anything real-time we highly suggest using the aggregation framework. Map/Reduce (v8 or not) should be used for offline processing only.&lt;/p&gt;</comment>
                            <comment id="245238" author="wwatson" created="Mon, 21 Jan 2013 02:52:18 +0000"  >&lt;p&gt;Well we&apos;ve just got one massive, unsharded server. We&apos;re not happy with it, but it works and it&apos;s fast for everything besides some map reduces since. You can close the ticket. I think we know what we need to know if we want to start back up a cluster. Thanks.&lt;/p&gt;</comment>
                            <comment id="244964" author="tyler@10gen.com" created="Sun, 20 Jan 2013 15:19:22 +0000"  >&lt;p&gt;As long as you are set as far as your cluster goes and you are happy with how things are working we should close this as it is a duplicate of the existing issue.&lt;/p&gt;</comment>
                            <comment id="244963" author="tyler@10gen.com" created="Sun, 20 Jan 2013 15:18:12 +0000"  >&lt;p&gt;We are on it William, it looks like the fix is scheduled for 2.5&lt;/p&gt;</comment>
                            <comment id="243866" author="tyler@10gen.com" created="Fri, 18 Jan 2013 20:49:35 +0000"  >&lt;p&gt;Linking well known problem with count on sharded collections.&lt;/p&gt;</comment>
                            <comment id="243862" author="wwatson" created="Fri, 18 Jan 2013 20:45:43 +0000"  >&lt;p&gt;Well this problem is not solved. We worked around it, but other people may have similar issues. I think this is worth investigating on your side and trying to replicate it.&lt;/p&gt;</comment>
                            <comment id="243850" author="tyler@10gen.com" created="Fri, 18 Jan 2013 20:35:51 +0000"  >&lt;p&gt;Ok, sounds good William. Should we close this ticket out?&lt;/p&gt;</comment>
                            <comment id="243740" author="wwatson" created="Fri, 18 Jan 2013 18:30:52 +0000"  >&lt;p&gt;Yeah, didn&apos;t get to really see the orphanage.js because we&apos;ve abandoned sharding in our main collections and I&apos;ve dropped the unsharded versions for disk space issues. I was able to run it and have it report no orphans on our sharded cluster. &lt;/p&gt;

&lt;p&gt;But thanks a bunch for giving us that. If we go back to sharding, I&apos;m sure it will come in handy.&lt;/p&gt;</comment>
                            <comment id="243561" author="wwatson" created="Fri, 18 Jan 2013 15:30:48 +0000"  >&lt;p&gt;I&apos;ll be running the orphan script today and I&apos;ll let you know how it goes. But just so you know, we moved off the replicated/sharded cluster yesterday because of the 4 or 5 issues we were seeing in Mongo 2.2.2. &lt;/p&gt;

&lt;p&gt;I have filed a bug for one of those issues now &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-8228&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-8228&lt;/a&gt; and I will continue to file other bugs we have seen. We&apos;re seeing a lot of data inconsistency that we can&apos;t have in our reports.&lt;/p&gt;</comment>
                            <comment id="242538" author="wwatson" created="Thu, 17 Jan 2013 15:08:00 +0000"  >&lt;p&gt;Hi Tyler, &lt;/p&gt;

&lt;p&gt;Thanks for that information. We&apos;ll let you know how the orphanage.js script works since we haven&apos;t run it yet. &lt;/p&gt;

&lt;p&gt;We were under the gun so we migrated the collections to non-sharded collections in a non-sharded database which seems to not have the same issues when we run map reduces into it.&lt;/p&gt;
</comment>
                            <comment id="242524" author="tyler@10gen.com" created="Thu, 17 Jan 2013 14:54:45 +0000"  >&lt;p&gt;Hey William,&lt;/p&gt;

&lt;p&gt;I work on maintaining the Ruby driver here and this is something that could happen if there are orphaned documents depending on how mongoid actually does the count and constructs the query.&lt;/p&gt;

&lt;p&gt;How did the orphanage.js script work for you?&lt;/p&gt;</comment>
                            <comment id="242076" author="wwatson" created="Thu, 17 Jan 2013 02:26:25 +0000"  >&lt;p&gt;Arron, I realize you normally only help with mongod-specific stuff, but this sort of applies. In the example I gave above:&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;2 documents returned by query&lt;/li&gt;
	&lt;li&gt;4 documents returned in count&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;The application layer (Rails/Mongoid/Mongo ruby gem) run that same query and returns 4 documents. The 2 that we cannot find when we query the mongod directly are older documents. Sorry if I say rows, don&apos;t mean to be inept, I just am. &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;</comment>
                            <comment id="241992" author="tyler@10gen.com" created="Wed, 16 Jan 2013 23:55:00 +0000"  >&lt;p&gt;William, to be sure, although its in the script startup messsage:&lt;/p&gt;

&lt;p&gt;1) This must be run on a mongos &lt;br/&gt;
2) You must follow the instructions to stop and ensure the balancer is stopped before using the find/remove methods. I believe it prevents you from using them without it being stopped but I wanted to make sure you knew that was the case.&lt;/p&gt;</comment>
                            <comment id="241981" author="aaron" created="Wed, 16 Jan 2013 23:46:59 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-5931&quot; title=&quot;Secondary reads in sharded clusters need stronger consistency&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-5931&quot;&gt;&lt;del&gt;SERVER-5931&lt;/del&gt;&lt;/a&gt; describes use of a non primary.  Are you using a non primary here?&lt;/p&gt;

&lt;p&gt;What do you mean when you say rows are being returned by your application layer?  What is your application layer doing?  Usually we can just help with mongod specific questions.&lt;/p&gt;</comment>
                            <comment id="241971" author="wwatson" created="Wed, 16 Jan 2013 23:41:08 +0000"  >&lt;p&gt;Thanks for that, it should be helpful. Now about my other couple of questions... is this exactly like issue &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-5931&quot; title=&quot;Secondary reads in sharded clusters need stronger consistency&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-5931&quot;&gt;&lt;del&gt;SERVER-5931&lt;/del&gt;&lt;/a&gt;? And could this be the cause of rows not being found in the database anywhere but old (possibly orphaned) rows with the same ID getting returned with the new rows from the application layer?&lt;/p&gt;</comment>
                            <comment id="241942" author="aaron" created="Wed, 16 Jan 2013 23:10:54 +0000"  >&lt;p&gt;Hi William,&lt;/p&gt;

&lt;p&gt;One of the 10gen engineers wrote a script to clean up these orphaned documents.  Be advised that the script is not officially supported - use at your own risk.  The script is supposed to print out what it is doing to make its behavior clear.  You can check the documents identified as orphaned before deleting them.&lt;/p&gt;

&lt;p&gt;The script has been attached to this ticket.&lt;/p&gt;

&lt;p&gt;You can import this script in the shell with&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;load(&quot;orphanage.js&quot;)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;According to the author:&lt;/p&gt;

&lt;p&gt;There are plenty of prompts so that you don&apos;t shoot yourself in the foot and know exactly what is being done. There is even an option for soft delete which simply adds a field _removed = true to the document so that the user can manually remove them later if they wish.&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;var result = Orphans.find(&apos;db.collection&apos;)&quot;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;result.next()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;// prints details about chunk and what would happen if he called remove&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;result.next()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Keep calling next() and looking at what would be removed. If you feel comfortable, you can use the remove functionality. The script is documented and prints usage on load.&lt;/p&gt;</comment>
                            <comment id="241882" author="wwatson" created="Wed, 16 Jan 2013 22:10:45 +0000"  >&lt;p&gt;Also, is this issue related to the one I just described? &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-5931&quot; title=&quot;Secondary reads in sharded clusters need stronger consistency&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-5931&quot;&gt;&lt;del&gt;SERVER-5931&lt;/del&gt;&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="241881" author="wwatson" created="Wed, 16 Jan 2013 22:09:36 +0000"  >&lt;p&gt;Is there something we can do to clean up the invalid chunks without having to reimport the data? We&apos;re also seeing that the rows represented by the above count getting returned from our application layer, but we cannot find them in the database anywhere.&lt;/p&gt;</comment>
                            <comment id="241862" author="aaron" created="Wed, 16 Jan 2013 21:42:02 +0000"  >&lt;p&gt;Hi William,&lt;/p&gt;

&lt;p&gt;Here is a likely cause of the issue you saw:&lt;/p&gt;

&lt;p&gt;Right now when a query runs, each matching document on a mongod is checked to see if it belongs to a chunk owned by that mongod.  If it does not belong to a chunk owned by the mongod, it is excluded from the returned results.  (A document not belonging to a mongod&apos;s set of chunks may exist on a mongod if 1) a migration is in progress or 2) an earlier migration was aborted, leaving orphaned documents on the mongod.)  The explain output format describes documents that are excluded because they do not belong to valid chunks (this is what the nChunkSkips attribute refers to).&lt;/p&gt;

&lt;p&gt;The count operation, unlike the query operation, does not check each matching document to see if it belongs to a valid chunk for the mongod.  It just counts all the matching documents.  This means in-migration or orphaned documents may be counted by the count operation even though they are not returned by the query operation.&lt;/p&gt;

&lt;p&gt;Based on your explain results, it looks like the problem may have gone away as a result of your re-import, which would have removed any orphaned documents.&lt;/p&gt;

&lt;p&gt;I believe the ticket representing this issue is &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-3645&quot; title=&quot;Sharded collection counts (on primary) can report too many results&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-3645&quot;&gt;&lt;del&gt;SERVER-3645&lt;/del&gt;&lt;/a&gt;, though this case may not be mentioned explicitly in the description.&lt;/p&gt;</comment>
                            <comment id="241049" author="wwatson" created="Wed, 16 Jan 2013 02:32:05 +0000"  >&lt;p&gt;Two things:&lt;/p&gt;

&lt;p&gt;1.) We found a work around a month ago and I just executed the work around after I put in this ticket b/c we desperately needed this collection be right today. The work around is to export and re-import the collection.&lt;/p&gt;

&lt;p&gt;2.) I hope my work around did not mess up this explain.&lt;/p&gt;

&lt;p&gt;{&lt;br/&gt;
        &quot;clusteredType&quot; : &quot;ParallelSort&quot;,&lt;br/&gt;
        &quot;shards&quot; : {&lt;br/&gt;
                &quot;domain_01/mongod01.internal.domain.net:90190,mongod02.internal.domain.net:90190&quot; : [&lt;br/&gt;
                        {&lt;br/&gt;
                                &quot;cursor&quot; : &quot;BasicCursor&quot;,&lt;br/&gt;
                                &quot;isMultiKey&quot; : false,&lt;br/&gt;
                                &quot;n&quot; : 0,&lt;br/&gt;
                                &quot;nscannedObjects&quot; : 6347226,&lt;br/&gt;
                                &quot;nscanned&quot; : 6347226,&lt;br/&gt;
                                &quot;nscannedObjectsAllPlans&quot; : 6347226,&lt;br/&gt;
                                &quot;nscannedAllPlans&quot; : 6347226,&lt;br/&gt;
                                &quot;scanAndOrder&quot; : false,&lt;br/&gt;
                                &quot;indexOnly&quot; : false,&lt;br/&gt;
                                &quot;nYields&quot; : 26,&lt;br/&gt;
                                &quot;nChunkSkips&quot; : 0,&lt;br/&gt;
                                &quot;millis&quot; : 25008,&lt;br/&gt;
                                &quot;indexBounds&quot; : {&lt;/p&gt;

&lt;p&gt;                                },&lt;br/&gt;
                                &quot;allPlans&quot; : [&lt;br/&gt;
                                        {&lt;br/&gt;
                                                &quot;cursor&quot; : &quot;BasicCursor&quot;,&lt;br/&gt;
                                                &quot;n&quot; : 0,&lt;br/&gt;
                                                &quot;nscannedObjects&quot; : 6347226,&lt;br/&gt;
                                                &quot;nscanned&quot; : 6347226,&lt;br/&gt;
                                                &quot;indexBounds&quot; : {&lt;/p&gt;

&lt;p&gt;                                                }&lt;br/&gt;
                                        }&lt;br/&gt;
                                ],&lt;br/&gt;
                                &quot;server&quot; : &quot;mongod02.domain.net:90190&quot;&lt;br/&gt;
                        }&lt;br/&gt;
                ],&lt;br/&gt;
                &quot;domain_02/mongod01.internal.domain.net:90290,mongod02.internal.domain.net:90290&quot; : [&lt;br/&gt;
                        {&lt;br/&gt;
                                &quot;cursor&quot; : &quot;BasicCursor&quot;,&lt;br/&gt;
                                &quot;isMultiKey&quot; : false,&lt;br/&gt;
                                &quot;n&quot; : 2,&lt;br/&gt;
                                &quot;nscannedObjects&quot; : 6303875,&lt;br/&gt;
                                &quot;nscanned&quot; : 6303875,&lt;br/&gt;
                                &quot;nscannedObjectsAllPlans&quot; : 6303875,&lt;br/&gt;
                                &quot;nscannedAllPlans&quot; : 6303875,&lt;br/&gt;
                                &quot;scanAndOrder&quot; : false,&lt;br/&gt;
                                &quot;indexOnly&quot; : false,&lt;br/&gt;
                                &quot;nYields&quot; : 32,&lt;br/&gt;
                                &quot;nChunkSkips&quot; : 0,&lt;br/&gt;
                                &quot;millis&quot; : 29651,&lt;br/&gt;
                                &quot;indexBounds&quot; : {&lt;/p&gt;

&lt;p&gt;                                },&lt;br/&gt;
                                &quot;allPlans&quot; : [&lt;br/&gt;
                                        {&lt;br/&gt;
                                                &quot;cursor&quot; : &quot;BasicCursor&quot;,&lt;br/&gt;
                                                &quot;n&quot; : 2,&lt;br/&gt;
                                                &quot;nscannedObjects&quot; : 6303875,&lt;br/&gt;
                                                &quot;nscanned&quot; : 6303875,&lt;br/&gt;
                                                &quot;indexBounds&quot; : {&lt;/p&gt;

&lt;p&gt;                                                }&lt;br/&gt;
                                        }&lt;br/&gt;
                                ],&lt;br/&gt;
                                &quot;server&quot; : &quot;mongod02.domain.net:90290&quot;&lt;br/&gt;
                        }&lt;br/&gt;
                ],&lt;br/&gt;
                &quot;domain_03/mongod01.internal.domain.net:90390,mongod02.internal.domain.net:90390&quot; : [&lt;br/&gt;
                        {&lt;br/&gt;
                                &quot;cursor&quot; : &quot;BasicCursor&quot;,&lt;br/&gt;
                                &quot;isMultiKey&quot; : false,&lt;br/&gt;
                                &quot;n&quot; : 0,&lt;br/&gt;
                                &quot;nscannedObjects&quot; : 6440956,&lt;br/&gt;
                                &quot;nscanned&quot; : 6440956,&lt;br/&gt;
                                &quot;nscannedObjectsAllPlans&quot; : 6440956,&lt;br/&gt;
                                &quot;nscannedAllPlans&quot; : 6440956,&lt;br/&gt;
                                &quot;scanAndOrder&quot; : false,&lt;br/&gt;
                                &quot;indexOnly&quot; : false,&lt;br/&gt;
                                &quot;nYields&quot; : 29,&lt;br/&gt;
                                &quot;nChunkSkips&quot; : 0,&lt;br/&gt;
                                &quot;millis&quot; : 25878,&lt;br/&gt;
                                &quot;indexBounds&quot; : {&lt;/p&gt;

&lt;p&gt;                                },&lt;br/&gt;
                                &quot;allPlans&quot; : [&lt;br/&gt;
                                        {&lt;br/&gt;
                                                &quot;cursor&quot; : &quot;BasicCursor&quot;,&lt;br/&gt;
                                                &quot;n&quot; : 0,&lt;br/&gt;
                                                &quot;nscannedObjects&quot; : 6440956,&lt;br/&gt;
                                                &quot;nscanned&quot; : 6440956,&lt;br/&gt;
                                                &quot;indexBounds&quot; : {&lt;/p&gt;

&lt;p&gt;                                                }&lt;br/&gt;
                                        }&lt;br/&gt;
                                ],&lt;br/&gt;
                                &quot;server&quot; : &quot;mongod02.domain.net:90390&quot;&lt;br/&gt;
                        }&lt;br/&gt;
                ],&lt;br/&gt;
                &quot;domain_04/mongod01.internal.domain.net:90490,mongod02.internal.domain.net:90490&quot; : [&lt;br/&gt;
                        {&lt;br/&gt;
                                &quot;cursor&quot; : &quot;BasicCursor&quot;,&lt;br/&gt;
                                &quot;isMultiKey&quot; : false,&lt;br/&gt;
                                &quot;n&quot; : 0,&lt;br/&gt;
                                &quot;nscannedObjects&quot; : 9789319,&lt;br/&gt;
                                &quot;nscanned&quot; : 9789319,&lt;br/&gt;
                                &quot;nscannedObjectsAllPlans&quot; : 9789319,&lt;br/&gt;
                                &quot;nscannedAllPlans&quot; : 9789319,&lt;br/&gt;
                                &quot;scanAndOrder&quot; : false,&lt;br/&gt;
                                &quot;indexOnly&quot; : false,&lt;br/&gt;
                                &quot;nYields&quot; : 92,&lt;br/&gt;
                                &quot;nChunkSkips&quot; : 0,&lt;br/&gt;
                                &quot;millis&quot; : 88849,&lt;br/&gt;
                                &quot;indexBounds&quot; : {&lt;/p&gt;

&lt;p&gt;                                },&lt;br/&gt;
                                &quot;allPlans&quot; : [&lt;br/&gt;
                                        {&lt;br/&gt;
                                                &quot;cursor&quot; : &quot;BasicCursor&quot;,&lt;br/&gt;
                                                &quot;n&quot; : 0,&lt;br/&gt;
                                                &quot;nscannedObjects&quot; : 9789319,&lt;br/&gt;
                                                &quot;nscanned&quot; : 9789319,&lt;br/&gt;
                                                &quot;indexBounds&quot; : {&lt;/p&gt;

&lt;p&gt;                                                }&lt;br/&gt;
                                        }&lt;br/&gt;
                                ],&lt;br/&gt;
                                &quot;server&quot; : &quot;mongod02.domain.net:90490&quot;&lt;br/&gt;
                        }&lt;br/&gt;
                ]&lt;br/&gt;
        },&lt;br/&gt;
        &quot;cursor&quot; : &quot;BasicCursor&quot;,&lt;br/&gt;
        &quot;n&quot; : 2,&lt;br/&gt;
        &quot;nChunkSkips&quot; : 0,&lt;br/&gt;
        &quot;nYields&quot; : 179,&lt;br/&gt;
        &quot;nscanned&quot; : 28881376,&lt;br/&gt;
        &quot;nscannedAllPlans&quot; : 28881376,&lt;br/&gt;
        &quot;nscannedObjects&quot; : 28881376,&lt;br/&gt;
        &quot;nscannedObjectsAllPlans&quot; : 28881376,&lt;br/&gt;
        &quot;millisShardTotal&quot; : 169386,&lt;br/&gt;
        &quot;millisShardAvg&quot; : 42346,&lt;br/&gt;
        &quot;numQueries&quot; : 4,&lt;br/&gt;
        &quot;numShards&quot; : 4,&lt;br/&gt;
        &quot;millis&quot; : 88942&lt;br/&gt;
}&lt;/p&gt;</comment>
                            <comment id="241013" author="aaron" created="Wed, 16 Jan 2013 01:37:12 +0000"  >&lt;p&gt;Hi William, could you send the output of a&lt;/p&gt;

&lt;p&gt;db.agg_collection.find(&lt;/p&gt;
{&quot;_id.summary_id&quot;: &quot;c48b9bb63&quot;, &quot;_id.placement_id&quot;: &quot;1&quot;}
&lt;p&gt;).explain( true )&lt;/p&gt;

&lt;p&gt;?&lt;/p&gt;</comment>
                            <comment id="241010" author="wwatson" created="Wed, 16 Jan 2013 01:32:55 +0000"  >&lt;p&gt;Also, it should be noted, we&apos;ve replicated this many times. &lt;/p&gt;

&lt;p&gt;Furthermore, we run many map reduces on other collections to generate the documents in the collections like the one above. I don&apos;t know if that helps debug the issue...&lt;/p&gt;</comment>
                            <comment id="241005" author="wwatson" created="Wed, 16 Jan 2013 01:27:52 +0000"  >&lt;p&gt;Hi Arron, no we did not. No exporting or importing was happening. &lt;/p&gt;</comment>
                            <comment id="240965" author="aaron" created="Wed, 16 Jan 2013 00:25:19 +0000"  >&lt;p&gt;Hi William - Did you have any migrations in progress when you ran this test?&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="21174">SERVER-3645</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="63754">SERVER-8405</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="23208" name="orphanage.js" size="11161" author="aaron" created="Wed, 16 Jan 2013 23:12:56 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>29.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[500A000000UaSUhIAN]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 16 Jan 2013 00:25:19 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        11 years, 3 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            11 years, 3 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>aaron</customfieldvalue>
            <customfieldvalue>tyler@10gen.com</customfieldvalue>
            <customfieldvalue>wwatson</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrnbmf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrllv3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>39588</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrkec7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>