[SERVER-8178] Odd Count / Document Results Differential On Sharded Collection Created: 15/Jan/13  Updated: 26/Sep/17  Resolved: 21/Jan/13

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 2.0.0, 2.2.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: William Watson Assignee: Aaron Staple
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File orphanage.js    
Issue Links:
Duplicate
duplicates SERVER-3645 Sharded collection counts (on primary... Closed
duplicates SERVER-8405 sharded count may incorrectly count m... Closed
Operating System: ALL
Participants:
Case:

 Description   

I get a count of 4 and a result of 2 documents when I do the following on a sharded collection

mongos> db.agg_collection.find(

{"_id.summary_id": "c48b9bb63", "_id.placement_id": "1"}

).count()
4
mongos> db.agg_collection.find(

{"_id.summary_id": "c48b9bb63", "_id.placement_id": "1"}

).pretty()
{
"_id" :

{ "placement_id" : "1", "summary_id" : "c48b9bb63", "new_or_used" : "N" }

,
"value" : 'hidden for patent reasons'
}
{
"_id" :

{ "placement_id" : "1", "summary_id" : "c48b9bb63", "new_or_used" : "U" }

,
"value" : 'hidden for patent reasons'
}



 Comments   
Comment by William Watson [ 30/Jan/13 ]

Yeah, that probably would explain it. We WERE running on a sharded cluster with slave_ok on, but we are now running everything on a single mongoD until we are confident everything is right again with sharding, but I just wanted to make sure the community gets the information they need.

Comment by Aaron Staple [ 30/Jan/13 ]

Hi William - Are you just seeing that behavior when you allow slave ok queries? If so, then SERVER-5931 is likely the cause. I can update the SERVER-5665 description to include the fact that it just describes the behavior on primaries, and SERVER-5931 represents addressing behavior on secondaries. If you do not have slave ok queries enabled, then we can help diagnose the situation if you can provide a precise description of what differentiates the application layer queries from the non application layer queries.

Comment by William Watson [ 30/Jan/13 ]

Hi Aaron,

I see your discussion on there. The phrase of note:

"When a query (not count) runs on a sharded cluster, each shard checks the documents that match the query to see if they also belong to chunks owned by that shard. If they do not belong to owned chunks, they are not returned."

This is not correct. We are seeing exactly that situation, that documents which match the query and do not belong to the shard are returned, but only by the application-layer Ruby MongoDB driver.

Comment by Aaron Staple [ 30/Jan/13 ]

Hi William - A plan for properly handling this case is being discussed internally. You can track our progress on implementing at SERVER-5665.

Comment by William Watson [ 29/Jan/13 ]

The aggregation framework won't work for us because what we're doing is pretty complex. However, I want to make myself clear about something that I don't think was really clear before.

Your database, through the mongoDB ruby connector, is returning incorrect data because of these orphan documents. This was a HUGE issue for us and I'm sure it is a huge issue for other companies as well. A database should not return incorrect data. Given that, I think your fixes for orphan documents should be patched into 2.2 as soon as humanly possible. Furthermore, I believe the ruby connector needs to be fixed since it was the connector returning the wrong documents. The command line returned the wrong counts which is also an issue.

Comment by Tyler Brock [ 21/Jan/13 ]

Ok,

Regarding slow map reduce, you can use the aggregation framework (2.2+) which avoids using the javascript engine (slow), allows for greater concurrency, and whose operations are expressed via a pipeline of json documents (easier to work with). Alternatively you may be very interested in Mongo 2.4 when released as we have switched the javascript engine to v8 from spider monkey (which also allows for greater concurrency). However, for anything real-time we highly suggest using the aggregation framework. Map/Reduce (v8 or not) should be used for offline processing only.

Comment by William Watson [ 21/Jan/13 ]

Well we've just got one massive, unsharded server. We're not happy with it, but it works and it's fast for everything besides some map reduces since. You can close the ticket. I think we know what we need to know if we want to start back up a cluster. Thanks.

Comment by Tyler Brock [ 20/Jan/13 ]

As long as you are set as far as your cluster goes and you are happy with how things are working we should close this as it is a duplicate of the existing issue.

Comment by Tyler Brock [ 20/Jan/13 ]

We are on it William, it looks like the fix is scheduled for 2.5

Comment by Tyler Brock [ 18/Jan/13 ]

Linking well known problem with count on sharded collections.

Comment by William Watson [ 18/Jan/13 ]

Well this problem is not solved. We worked around it, but other people may have similar issues. I think this is worth investigating on your side and trying to replicate it.

Comment by Tyler Brock [ 18/Jan/13 ]

Ok, sounds good William. Should we close this ticket out?

Comment by William Watson [ 18/Jan/13 ]

Yeah, didn't get to really see the orphanage.js because we've abandoned sharding in our main collections and I've dropped the unsharded versions for disk space issues. I was able to run it and have it report no orphans on our sharded cluster.

But thanks a bunch for giving us that. If we go back to sharding, I'm sure it will come in handy.

Comment by William Watson [ 18/Jan/13 ]

I'll be running the orphan script today and I'll let you know how it goes. But just so you know, we moved off the replicated/sharded cluster yesterday because of the 4 or 5 issues we were seeing in Mongo 2.2.2.

I have filed a bug for one of those issues now https://jira.mongodb.org/browse/SERVER-8228 and I will continue to file other bugs we have seen. We're seeing a lot of data inconsistency that we can't have in our reports.

Comment by William Watson [ 17/Jan/13 ]

Hi Tyler,

Thanks for that information. We'll let you know how the orphanage.js script works since we haven't run it yet.

We were under the gun so we migrated the collections to non-sharded collections in a non-sharded database which seems to not have the same issues when we run map reduces into it.

Comment by Tyler Brock [ 17/Jan/13 ]

Hey William,

I work on maintaining the Ruby driver here and this is something that could happen if there are orphaned documents depending on how mongoid actually does the count and constructs the query.

How did the orphanage.js script work for you?

Comment by William Watson [ 17/Jan/13 ]

Arron, I realize you normally only help with mongod-specific stuff, but this sort of applies. In the example I gave above:

  • 2 documents returned by query
  • 4 documents returned in count

The application layer (Rails/Mongoid/Mongo ruby gem) run that same query and returns 4 documents. The 2 that we cannot find when we query the mongod directly are older documents. Sorry if I say rows, don't mean to be inept, I just am.

Comment by Tyler Brock [ 16/Jan/13 ]

William, to be sure, although its in the script startup messsage:

1) This must be run on a mongos
2) You must follow the instructions to stop and ensure the balancer is stopped before using the find/remove methods. I believe it prevents you from using them without it being stopped but I wanted to make sure you knew that was the case.

Comment by Aaron Staple [ 16/Jan/13 ]

SERVER-5931 describes use of a non primary. Are you using a non primary here?

What do you mean when you say rows are being returned by your application layer? What is your application layer doing? Usually we can just help with mongod specific questions.

Comment by William Watson [ 16/Jan/13 ]

Thanks for that, it should be helpful. Now about my other couple of questions... is this exactly like issue SERVER-5931? And could this be the cause of rows not being found in the database anywhere but old (possibly orphaned) rows with the same ID getting returned with the new rows from the application layer?

Comment by Aaron Staple [ 16/Jan/13 ]

Hi William,

One of the 10gen engineers wrote a script to clean up these orphaned documents. Be advised that the script is not officially supported - use at your own risk. The script is supposed to print out what it is doing to make its behavior clear. You can check the documents identified as orphaned before deleting them.

The script has been attached to this ticket.

You can import this script in the shell with

load("orphanage.js")

According to the author:

There are plenty of prompts so that you don't shoot yourself in the foot and know exactly what is being done. There is even an option for soft delete which simply adds a field _removed = true to the document so that the user can manually remove them later if they wish.

var result = Orphans.find('db.collection')"
result.next()
// prints details about chunk and what would happen if he called remove
result.next()

Keep calling next() and looking at what would be removed. If you feel comfortable, you can use the remove functionality. The script is documented and prints usage on load.

Comment by William Watson [ 16/Jan/13 ]

Also, is this issue related to the one I just described? SERVER-5931

Comment by William Watson [ 16/Jan/13 ]

Is there something we can do to clean up the invalid chunks without having to reimport the data? We're also seeing that the rows represented by the above count getting returned from our application layer, but we cannot find them in the database anywhere.

Comment by Aaron Staple [ 16/Jan/13 ]

Hi William,

Here is a likely cause of the issue you saw:

Right now when a query runs, each matching document on a mongod is checked to see if it belongs to a chunk owned by that mongod. If it does not belong to a chunk owned by the mongod, it is excluded from the returned results. (A document not belonging to a mongod's set of chunks may exist on a mongod if 1) a migration is in progress or 2) an earlier migration was aborted, leaving orphaned documents on the mongod.) The explain output format describes documents that are excluded because they do not belong to valid chunks (this is what the nChunkSkips attribute refers to).

The count operation, unlike the query operation, does not check each matching document to see if it belongs to a valid chunk for the mongod. It just counts all the matching documents. This means in-migration or orphaned documents may be counted by the count operation even though they are not returned by the query operation.

Based on your explain results, it looks like the problem may have gone away as a result of your re-import, which would have removed any orphaned documents.

I believe the ticket representing this issue is SERVER-3645, though this case may not be mentioned explicitly in the description.

Comment by William Watson [ 16/Jan/13 ]

Two things:

1.) We found a work around a month ago and I just executed the work around after I put in this ticket b/c we desperately needed this collection be right today. The work around is to export and re-import the collection.

2.) I hope my work around did not mess up this explain.

{
"clusteredType" : "ParallelSort",
"shards" : {
"domain_01/mongod01.internal.domain.net:90190,mongod02.internal.domain.net:90190" : [
{
"cursor" : "BasicCursor",
"isMultiKey" : false,
"n" : 0,
"nscannedObjects" : 6347226,
"nscanned" : 6347226,
"nscannedObjectsAllPlans" : 6347226,
"nscannedAllPlans" : 6347226,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 26,
"nChunkSkips" : 0,
"millis" : 25008,
"indexBounds" : {

},
"allPlans" : [
{
"cursor" : "BasicCursor",
"n" : 0,
"nscannedObjects" : 6347226,
"nscanned" : 6347226,
"indexBounds" : {

}
}
],
"server" : "mongod02.domain.net:90190"
}
],
"domain_02/mongod01.internal.domain.net:90290,mongod02.internal.domain.net:90290" : [
{
"cursor" : "BasicCursor",
"isMultiKey" : false,
"n" : 2,
"nscannedObjects" : 6303875,
"nscanned" : 6303875,
"nscannedObjectsAllPlans" : 6303875,
"nscannedAllPlans" : 6303875,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 32,
"nChunkSkips" : 0,
"millis" : 29651,
"indexBounds" : {

},
"allPlans" : [
{
"cursor" : "BasicCursor",
"n" : 2,
"nscannedObjects" : 6303875,
"nscanned" : 6303875,
"indexBounds" : {

}
}
],
"server" : "mongod02.domain.net:90290"
}
],
"domain_03/mongod01.internal.domain.net:90390,mongod02.internal.domain.net:90390" : [
{
"cursor" : "BasicCursor",
"isMultiKey" : false,
"n" : 0,
"nscannedObjects" : 6440956,
"nscanned" : 6440956,
"nscannedObjectsAllPlans" : 6440956,
"nscannedAllPlans" : 6440956,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 29,
"nChunkSkips" : 0,
"millis" : 25878,
"indexBounds" : {

},
"allPlans" : [
{
"cursor" : "BasicCursor",
"n" : 0,
"nscannedObjects" : 6440956,
"nscanned" : 6440956,
"indexBounds" : {

}
}
],
"server" : "mongod02.domain.net:90390"
}
],
"domain_04/mongod01.internal.domain.net:90490,mongod02.internal.domain.net:90490" : [
{
"cursor" : "BasicCursor",
"isMultiKey" : false,
"n" : 0,
"nscannedObjects" : 9789319,
"nscanned" : 9789319,
"nscannedObjectsAllPlans" : 9789319,
"nscannedAllPlans" : 9789319,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 92,
"nChunkSkips" : 0,
"millis" : 88849,
"indexBounds" : {

},
"allPlans" : [
{
"cursor" : "BasicCursor",
"n" : 0,
"nscannedObjects" : 9789319,
"nscanned" : 9789319,
"indexBounds" : {

}
}
],
"server" : "mongod02.domain.net:90490"
}
]
},
"cursor" : "BasicCursor",
"n" : 2,
"nChunkSkips" : 0,
"nYields" : 179,
"nscanned" : 28881376,
"nscannedAllPlans" : 28881376,
"nscannedObjects" : 28881376,
"nscannedObjectsAllPlans" : 28881376,
"millisShardTotal" : 169386,
"millisShardAvg" : 42346,
"numQueries" : 4,
"numShards" : 4,
"millis" : 88942
}

Comment by Aaron Staple [ 16/Jan/13 ]

Hi William, could you send the output of a

db.agg_collection.find(

{"_id.summary_id": "c48b9bb63", "_id.placement_id": "1"}

).explain( true )

?

Comment by William Watson [ 16/Jan/13 ]

Also, it should be noted, we've replicated this many times.

Furthermore, we run many map reduces on other collections to generate the documents in the collections like the one above. I don't know if that helps debug the issue...

Comment by William Watson [ 16/Jan/13 ]

Hi Arron, no we did not. No exporting or importing was happening.

Comment by Aaron Staple [ 16/Jan/13 ]

Hi William - Did you have any migrations in progress when you ran this test?

Generated at Thu Feb 08 03:16:46 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.