[SERVER-8178] Odd Count / Document Results Differential On Sharded Collection Created: 15/Jan/13 Updated: 26/Sep/17 Resolved: 21/Jan/13 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 2.0.0, 2.2.0 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | William Watson | Assignee: | Aaron Staple |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||
| Issue Links: |
|
||||||||||||
| Operating System: | ALL | ||||||||||||
| Participants: | |||||||||||||
| Case: | (copied to CRM) | ||||||||||||
| Description |
|
I get a count of 4 and a result of 2 documents when I do the following on a sharded collection mongos> db.agg_collection.find( {"_id.summary_id": "c48b9bb63", "_id.placement_id": "1"}).count() ).pretty() , , |
| Comments |
| Comment by William Watson [ 30/Jan/13 ] | |||||
|
Yeah, that probably would explain it. We WERE running on a sharded cluster with slave_ok on, but we are now running everything on a single mongoD until we are confident everything is right again with sharding, but I just wanted to make sure the community gets the information they need. | |||||
| Comment by Aaron Staple [ 30/Jan/13 ] | |||||
|
Hi William - Are you just seeing that behavior when you allow slave ok queries? If so, then | |||||
| Comment by William Watson [ 30/Jan/13 ] | |||||
|
Hi Aaron, I see your discussion on there. The phrase of note: "When a query (not count) runs on a sharded cluster, each shard checks the documents that match the query to see if they also belong to chunks owned by that shard. If they do not belong to owned chunks, they are not returned." This is not correct. We are seeing exactly that situation, that documents which match the query and do not belong to the shard are returned, but only by the application-layer Ruby MongoDB driver. | |||||
| Comment by Aaron Staple [ 30/Jan/13 ] | |||||
|
Hi William - A plan for properly handling this case is being discussed internally. You can track our progress on implementing at | |||||
| Comment by William Watson [ 29/Jan/13 ] | |||||
|
The aggregation framework won't work for us because what we're doing is pretty complex. However, I want to make myself clear about something that I don't think was really clear before. Your database, through the mongoDB ruby connector, is returning incorrect data because of these orphan documents. This was a HUGE issue for us and I'm sure it is a huge issue for other companies as well. A database should not return incorrect data. Given that, I think your fixes for orphan documents should be patched into 2.2 as soon as humanly possible. Furthermore, I believe the ruby connector needs to be fixed since it was the connector returning the wrong documents. The command line returned the wrong counts which is also an issue. | |||||
| Comment by Tyler Brock [ 21/Jan/13 ] | |||||
|
Ok, Regarding slow map reduce, you can use the aggregation framework (2.2+) which avoids using the javascript engine (slow), allows for greater concurrency, and whose operations are expressed via a pipeline of json documents (easier to work with). Alternatively you may be very interested in Mongo 2.4 when released as we have switched the javascript engine to v8 from spider monkey (which also allows for greater concurrency). However, for anything real-time we highly suggest using the aggregation framework. Map/Reduce (v8 or not) should be used for offline processing only. | |||||
| Comment by William Watson [ 21/Jan/13 ] | |||||
|
Well we've just got one massive, unsharded server. We're not happy with it, but it works and it's fast for everything besides some map reduces since. You can close the ticket. I think we know what we need to know if we want to start back up a cluster. Thanks. | |||||
| Comment by Tyler Brock [ 20/Jan/13 ] | |||||
|
As long as you are set as far as your cluster goes and you are happy with how things are working we should close this as it is a duplicate of the existing issue. | |||||
| Comment by Tyler Brock [ 20/Jan/13 ] | |||||
|
We are on it William, it looks like the fix is scheduled for 2.5 | |||||
| Comment by Tyler Brock [ 18/Jan/13 ] | |||||
|
Linking well known problem with count on sharded collections. | |||||
| Comment by William Watson [ 18/Jan/13 ] | |||||
|
Well this problem is not solved. We worked around it, but other people may have similar issues. I think this is worth investigating on your side and trying to replicate it. | |||||
| Comment by Tyler Brock [ 18/Jan/13 ] | |||||
|
Ok, sounds good William. Should we close this ticket out? | |||||
| Comment by William Watson [ 18/Jan/13 ] | |||||
|
Yeah, didn't get to really see the orphanage.js because we've abandoned sharding in our main collections and I've dropped the unsharded versions for disk space issues. I was able to run it and have it report no orphans on our sharded cluster. But thanks a bunch for giving us that. If we go back to sharding, I'm sure it will come in handy. | |||||
| Comment by William Watson [ 18/Jan/13 ] | |||||
|
I'll be running the orphan script today and I'll let you know how it goes. But just so you know, we moved off the replicated/sharded cluster yesterday because of the 4 or 5 issues we were seeing in Mongo 2.2.2. I have filed a bug for one of those issues now https://jira.mongodb.org/browse/SERVER-8228 and I will continue to file other bugs we have seen. We're seeing a lot of data inconsistency that we can't have in our reports. | |||||
| Comment by William Watson [ 17/Jan/13 ] | |||||
|
Hi Tyler, Thanks for that information. We'll let you know how the orphanage.js script works since we haven't run it yet. We were under the gun so we migrated the collections to non-sharded collections in a non-sharded database which seems to not have the same issues when we run map reduces into it. | |||||
| Comment by Tyler Brock [ 17/Jan/13 ] | |||||
|
Hey William, I work on maintaining the Ruby driver here and this is something that could happen if there are orphaned documents depending on how mongoid actually does the count and constructs the query. How did the orphanage.js script work for you? | |||||
| Comment by William Watson [ 17/Jan/13 ] | |||||
|
Arron, I realize you normally only help with mongod-specific stuff, but this sort of applies. In the example I gave above:
The application layer (Rails/Mongoid/Mongo ruby gem) run that same query and returns 4 documents. The 2 that we cannot find when we query the mongod directly are older documents. Sorry if I say rows, don't mean to be inept, I just am. | |||||
| Comment by Tyler Brock [ 16/Jan/13 ] | |||||
|
William, to be sure, although its in the script startup messsage: 1) This must be run on a mongos | |||||
| Comment by Aaron Staple [ 16/Jan/13 ] | |||||
|
What do you mean when you say rows are being returned by your application layer? What is your application layer doing? Usually we can just help with mongod specific questions. | |||||
| Comment by William Watson [ 16/Jan/13 ] | |||||
|
Thanks for that, it should be helpful. Now about my other couple of questions... is this exactly like issue | |||||
| Comment by Aaron Staple [ 16/Jan/13 ] | |||||
|
Hi William, One of the 10gen engineers wrote a script to clean up these orphaned documents. Be advised that the script is not officially supported - use at your own risk. The script is supposed to print out what it is doing to make its behavior clear. You can check the documents identified as orphaned before deleting them. The script has been attached to this ticket. You can import this script in the shell with
According to the author: There are plenty of prompts so that you don't shoot yourself in the foot and know exactly what is being done. There is even an option for soft delete which simply adds a field _removed = true to the document so that the user can manually remove them later if they wish.
Keep calling next() and looking at what would be removed. If you feel comfortable, you can use the remove functionality. The script is documented and prints usage on load. | |||||
| Comment by William Watson [ 16/Jan/13 ] | |||||
|
Also, is this issue related to the one I just described? | |||||
| Comment by William Watson [ 16/Jan/13 ] | |||||
|
Is there something we can do to clean up the invalid chunks without having to reimport the data? We're also seeing that the rows represented by the above count getting returned from our application layer, but we cannot find them in the database anywhere. | |||||
| Comment by Aaron Staple [ 16/Jan/13 ] | |||||
|
Hi William, Here is a likely cause of the issue you saw: Right now when a query runs, each matching document on a mongod is checked to see if it belongs to a chunk owned by that mongod. If it does not belong to a chunk owned by the mongod, it is excluded from the returned results. (A document not belonging to a mongod's set of chunks may exist on a mongod if 1) a migration is in progress or 2) an earlier migration was aborted, leaving orphaned documents on the mongod.) The explain output format describes documents that are excluded because they do not belong to valid chunks (this is what the nChunkSkips attribute refers to). The count operation, unlike the query operation, does not check each matching document to see if it belongs to a valid chunk for the mongod. It just counts all the matching documents. This means in-migration or orphaned documents may be counted by the count operation even though they are not returned by the query operation. Based on your explain results, it looks like the problem may have gone away as a result of your re-import, which would have removed any orphaned documents. I believe the ticket representing this issue is | |||||
| Comment by William Watson [ 16/Jan/13 ] | |||||
|
Two things: 1.) We found a work around a month ago and I just executed the work around after I put in this ticket b/c we desperately needed this collection be right today. The work around is to export and re-import the collection. 2.) I hope my work around did not mess up this explain. { }, } }, } }, } }, } | |||||
| Comment by Aaron Staple [ 16/Jan/13 ] | |||||
|
Hi William, could you send the output of a db.agg_collection.find( {"_id.summary_id": "c48b9bb63", "_id.placement_id": "1"}).explain( true ) ? | |||||
| Comment by William Watson [ 16/Jan/13 ] | |||||
|
Also, it should be noted, we've replicated this many times. Furthermore, we run many map reduces on other collections to generate the documents in the collections like the one above. I don't know if that helps debug the issue... | |||||
| Comment by William Watson [ 16/Jan/13 ] | |||||
|
Hi Arron, no we did not. No exporting or importing was happening. | |||||
| Comment by Aaron Staple [ 16/Jan/13 ] | |||||
|
Hi William - Did you have any migrations in progress when you ran this test? |