[SERVER-19603] Non-Sharded database is present in two different shards (with different data) Created: 27/Jul/15  Updated: 10/Aug/15  Resolved: 10/Aug/15

Status: Closed
Project: Core Server
Component/s: Admin, Sharding
Affects Version/s: 2.4.10
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Bruno Melo Assignee: Ramon Fernandez Marina
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-17397 Dropping a Database or Collection in ... Closed
Operating System: ALL
Participants:

 Description   

We have a production cluster (2.4.10) composed of 3 shards. Each shard is a replica set with 2 MongoD and 1 Arbiter. There are also 2 MongoS that provide access to the cluster.
A few weeks back we had to rebuild one of our databases to support an application version upgrade.
The procedure was to drop the database and let the application recreate and repopulate a database with the same name.
Afterwards we noticed inconsistencies in the results provided by the application, digging deeper we realized that the database which is not sharded was showing in 2 replica-sets:

mongo-rs-1-1:PRIMARY> show dbs
(...)
VODRepository_UNIFIED   0.453125GB
(...)
mongo-rs-1-1:PRIMARY>
 
mongo-rs-1-3:PRIMARY> show dbs
(...)
VODRepository_UNIFIED   0.453125GB
(...)
mongo-rs-1-3:PRIMARY>

Although both mongo-s instances showed the database in the same shard:

{  "_id" : "VODRepository_UNIFIED",  "partitioned" : false,  "primary" : "mongo-rs-1-3" }

Performing an explain on each node yields different results:

Mongo-s-1-1:

 
mongos> db.TitleAsset.find({ "MovieVideoStreams.AdiAssetId" : "CIMV624197D9C61F4806" }).limit(50).explain();
{
        "cursor" : "BasicCursor",
        "isMultiKey" : false,
        "n" : 1,
        "nscannedObjects" : 12455,
        "nscanned" : 12455,
        "nscannedObjectsAllPlans" : 12455,
        "nscannedAllPlans" : 12455,
        "scanAndOrder" : false,
        "indexOnly" : false,
        "nYields" : 0,
        "nChunkSkips" : 0,
        "millis" : 44,
        "indexBounds" : {
 
        },
        "server" : "SVLMNIPDB16.mng.hdi.tvcabo:27018",
        "millis" : 44
}
mongos>

Mongo-s-1-2:

 
mongos> db.TitleAsset.find({ "MovieVideoStreams.AdiAssetId" : "CIMV624197D9C61F4806" }).limit(50).explain();
{
        "cursor" : "BasicCursor",
        "isMultiKey" : false,
        "n" : 1,
        "nscannedObjects" : 12452,
        "nscanned" : 12452,
        "nscannedObjectsAllPlans" : 12452,
        "nscannedAllPlans" : 12452,
        "scanAndOrder" : false,
        "indexOnly" : false,
        "nYields" : 0,
        "nChunkSkips" : 0,
        "millis" : 44,
        "indexBounds" : {
 
        },
        "server" : "SVLMNIPDB12.mng.hdi.tvcabo:27018",
        "millis" : 44
}
mongos>



 Comments   
Comment by Ramon Fernandez Marina [ 10/Aug/15 ]

bruno.m.melo@nos.pt, have you had a chance to look into SERVER-17397 and fix the data duplication?

I'm going to mark this ticket as a duplicate of SERVER-17397, but feel free to post any updates to this ticket if you think they may benefit other users running into this issue. If you provide any additional information that may indicate that this is a different issue than SERVER-17397 we can always reopen this ticket and investigate further.

Regards,
Ramón.

Comment by Ramon Fernandez Marina [ 28/Jul/15 ]

Hi bruno.m.melo@nos.pt, I think you may be running into SERVER-17397, where dropping a database may not fully succeed and attempting to re-use the same namespace may lead to undefined behavior.

I'd suggest you back up both databases and go through the steps in the WORKAROUNDS section in the summary box for SERVER-17397; this should eliminate all traces of these collections in your cluster. After that it's safe to re-create the namespaces you need and restore the data.

Generated at Thu Feb 08 03:51:30 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.