-
Type:
Bug
-
Resolution: Community Answered
-
Priority:
Major - P3
-
None
-
Affects Version/s: 4.0.4
-
Component/s: Sharding
-
Labels:None
-
ALL
-
Hello, this is continuation of SERVER-44143 (the same environment)
We are getting the following error on our PROD sharded cluster (which was migrated from docker to other servers (withouth using the docker technology) by following the "Restore a Sharded Cluster" https://docs.mongodb.com/v4.0/tutorial/restore-sharded-cluster/#d-restore-each-shard-replica-set (with mongodump/mongorestore)
===========================================
2020-01-25T14:13:31.655+0100 I SHARDING [migrateThread] migrate failed: InvalidUUID: Cannot create collection productrepository.products because we already have an identically named collection with UUID 55ab81fa-7d21-4742-8d71-f4ef8f741ec2, which differs from the donor's UUID 3db9aaae-c037-4162-b0a8-9eec312df936. Manually drop the collection on this shard if it contains data from a previous incarnation of productrepository.products '
===========================================
Here is the collections UUIDs we get if we connect to each shard:
PROD:
shard1:PRIMARY> db.getCollectionInfos()
[
{
"name" : "products",
"type" : "collection",
"options" : {
},
"info" : {
"readOnly" : false,
"uuid" : UUID("b8dd9615-f861-4535-a434-9638f5e5c452")
},
"idIndex" : {
"v" : 2,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "productrepository.products"
}
},
shard2:PRIMARY> db.getCollectionInfos()
{
"name" : "products",
"type" : "collection",
"options" : {
},
"info" : {
"readOnly" : false,
"uuid" : UUID("55ab81fa-7d21-4742-8d71-f4ef8f741ec2")
},
"idIndex" : {
"v" : 2,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "productrepository.products"
}
},
shard3:PRIMARY> db.getCollectionInfos()
[
{
"name" : "products",
"type" : "collection",
"options" : {
},
"info" : {
"readOnly" : false,
"uuid" : UUID("3db9aaae-c037-4162-b0a8-9eec312df936")
},
"idIndex" : {
"v" : 2,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "productrepository.products"
}
},
Here we can see that UUID of the collection products in config replica set is the same as it is on shards in config.cache.collections
configserver:PRIMARY> db.collections.find()
{ "_id" : "config.system.sessions", "lastmodEpoch" : ObjectId("5bb4b070aec28d86b2174284"), "lastmod" : ISODate("1970-02-19T17:02:47.296Z"), "dropped" : false, "key" : { "_id" : 1 }, "unique" : false, "uuid" : UUID("68a953b8-1136-4891-ac27-acac48925d00") }
{ "_id" : "productrepository.products", "lastmodEpoch" : ObjectId("5bb4b060aec28d86b2174007"), "lastmod" : ISODate("1970-02-19T17:02:47.298Z"), "dropped" : false, "key" : { "productId" : "hashed" }, "unique" : false, "uuid" : UUID("0024b33d-2295-45b7-8bd1-d8ffb58a438d") }
shard1:PRIMARY> db.cache.collections.find()
{ "_id" : "config.system.sessions", "epoch" : ObjectId("5bb4b070aec28d86b2174284"), "key" : { "_id" : 1 }, "refreshing" : false, "unique" : false, "uuid" : UUID("68a953b8-1136-4891-ac27-acac48925d00"), "lastRefreshedCollectionVersion" : Timestamp(1, 0) }
{ "_id" : "productrepository.products", "epoch" : ObjectId("5bb4b060aec28d86b2174007"), "key" : { "productId" : "hashed" }, "refreshing" : false, "unique" : false, "uuid" : UUID("0024b33d-2295-45b7-8bd1-d8ffb58a438d"), "lastRefreshedCollectionVersion" : Timestamp(42, 91), "enterCriticalSectionCounter" : 13 }
shard2:PRIMARY> db.cache.collections.find()
{ "_id" : "config.system.sessions", "epoch" : ObjectId("5bb4b070aec28d86b2174284"), "key" : { "_id" : 1 }, "refreshing" : false, "unique" : false, "uuid" : UUID("68a953b8-1136-4891-ac27-acac48925d00"), "lastRefreshedCollectionVersion" : Timestamp(1, 0) }
{ "_id" : "productrepository.products", "epoch" : ObjectId("5bb4b060aec28d86b2174007"), "key" : { "productId" : "hashed" }, "refreshing" : false, "unique" : false, "uuid" : UUID("0024b33d-2295-45b7-8bd1-d8ffb58a438d"), "lastRefreshedCollectionVersion" : Timestamp(42, 91), "enterCriticalSectionCounter" : 17 }
shard3:PRIMARY> db.cache.collections.find()
{ "_id" : "config.system.sessions", "epoch" : ObjectId("5bb4b070aec28d86b2174284"), "key" : { "_id" : 1 }, "refreshing" : false, "unique" : false, "uuid" : UUID("68a953b8-1136-4891-ac27-acac48925d00"), "lastRefreshedCollectionVersion" : Timestamp(1, 0) }
{ "_id" : "productrepository.products", "epoch" : ObjectId("5bb4b060aec28d86b2174007"), "key" : { "productId" : "hashed" }, "refreshing" : false, "unique" : false, "uuid" : UUID("0024b33d-2295-45b7-8bd1-d8ffb58a438d"), "lastRefreshedCollectionVersion" : Timestamp(42, 91), "enterCriticalSectionCounter" : 11 }
The thing is that we have 2 TEST environmets (sharded clusters) that we clone our PROD to, every week. (We are restoring PROD's backup to the 2 TEST clusters, following the same procedure stated above)'
And I see, that on those two (cloned every week) environmets the UUID of products collection (productrepository.products) is every time unique and different between shards, as if mongorestore when we restore sequentially shards assigns new UUID to the sharded collection on each shard.
TEST cluster 1 :
shard1:PRIMARY> db.getCollectionInfos()
[
{
"name" : "products",
"type" : "collection",
"options" : {
},
"info" : {
"readOnly" : false,
"uuid" : UUID("aeffed7c-f1b0-453c-9614-1b42a70991ef")
},
"idIndex" : {
"v" : 2,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "productrepository.products"
}
shard2:PRIMARY> db.getCollectionInfos()
{
"name" : "products",
"type" : "collection",
"options" : {
},
"info" : {
"readOnly" : false,
"uuid" : UUID("e0cc3f43-ffec-4dd7-a67c-6e38b9635c7a")
},
"idIndex" : {
"v" : 2,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "productrepository.products"
}
},
shard3:PRIMARY> db.getCollectionInfos()
[
{
"name" : "products",
"type" : "collection",
"options" : {
},
"info" : {
"readOnly" : false,
"uuid" : UUID("95694fff-8918-4c23-900e-99a110476b0c")
},
"idIndex" : {
"v" : 2,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "productrepository.products"
}
},
TEST cluster №2
shard1:PRIMARY> db.getCollectionInfos()
[
{
"name" : "products",
"type" : "collection",
"options" : {
},
"info" : {
"readOnly" : false,
"uuid" : UUID("37823188-b8f6-4a8d-9a22-dd609d54302e")
},
"idIndex" : {
"v" : 2,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "productrepository.products"
}
}
shard2:PRIMARY> db.getCollectionInfos()
{
"name" : "products",
"type" : "collection",
"options" : {
},
"info" : {
"readOnly" : false,
"uuid" : UUID("b4de5c76-90f2-4075-9aa1-f6a99cef5608")
},
"idIndex" : {
"v" : 2,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "productrepository.products"
}
}
]
shard3:PRIMARY> db.getCollectionInfos()
[
{
"name" : "products",
"type" : "collection",
"options" : {
},
"info" : {
"readOnly" : false,
"uuid" : UUID("f43c6927-b6b4-4bd9-8e81-6e69478c5823")
},
"idIndex" : {
"v" : 2,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "productrepository.products"
}
}
]
I have tried manually moving chunks in TEST env, using moveChunk command, and as expected, getting the issue with different UUIDs
Is mongorestore supposed to assign new UUIDs to restored sharded collections?
As I understand, to rectify the issue with chunk migration, the only way is to drop the collection through mongos (following the procedure described in SERVER-17397), then restore the collection through mongos and shard it.
But if, in case of a disaster, we are forced to restore the PROD cluster from backup, we would have to recreate all sharded collections this way (so far we have only one, but that will change in the future), and also rework our clone procedure to recreate the sharded collections this way?
P.S. After another clone that took place some hours later after writing the information above, I checked once againg the UUID of the sharded collection, and again it is different between shards and also don't correlate to the UUIDs from PROD
Thank you.