-
Type: Bug
-
Resolution: Incomplete
-
Priority: Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Sharding
-
None
-
Fully Compatible
-
ALL
Hello MongoDB Team.
We faced with the issue related to moveChunk process.
This is the error from one of the Shrad's log:
2019-10-22T11:18:45.921+0200 I SHARDING [conn132] Starting chunk migration ns: db.col, [{ productId: MinKey }, { productId: -9062096908397525382 }), fromShard: shard3, toShard: shard2 with expected collection version epoch 5bb4b060aec28d86b2174007 2019-10-22T11:18:45.924+0200 I SHARDING [conn132] about to log metadata event into changelog: { _id: "prulmcoembb09-2019-10-22T11:18:45.924+0200-5daec975680ef73c96f4be36", server: "prulmcoembb09", clientAddr: "10.217.1.252:50292", time: new Date(1571735925924), what: "moveChunk.start", ns: "db.col", details: { min: { productId: MinKey }, max: { productId: -9062096908397525382 }, from: "shard3", to: "shard2" } } 2019-10-22T11:18:45.955+0200 I SHARDING [conn132] moveChunk data transfer progress: { waited: true, active: false, ns: "db.col", from: "shard3/IP:PORT", fromShardId: "shard3", min: { productId: MinKey }, max: { productId: -9062096908397525382 }, shardKeyPattern: { productId: "hashed" }, state: "fail", errmsg: "migrate failed: InvalidUUID: Cannot create collection db.col because we already have an identically named collection with UUID 55a...", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0, operationTime: Timestamp(1571735921, 1), $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff000000000000001f') }, lastCommittedOpTime: Timestamp(1571735921, 1), $configServerState: { opTime: { ts: Timestamp(1571735925, 6), t: 18 } }, $clusterTime: { clusterTime: Timestamp(1571735925, 6), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } } } mem used: 0 documents remaining to clone: 2731 2019-10-22T11:18:45.955+0200 I SHARDING [conn132] about to log metadata event into changelog: { _id: "prulmcoembb09-2019-10-22T11:18:45.955+0200-5daec975680ef73c96f4be5d", server: "prulmcoembb09", clientAddr: "10.217.1.252:50292", time: new Date(1571735925955), what: "moveChunk.error", ns: "db.col", details: { min: { productId: MinKey }, max: { productId: -9062096908397525382 }, from: "shard3", to: "shard2" } } 2019-10-22T11:18:45.961+0200 W SHARDING [conn132] Chunk move failed :: caused by :: OperationFailed: Data transfer error: migrate failed: InvalidUUID: Cannot create collection db.col because we already have an identically named collection with UUID 55ab81fa-7d21-4742-8d71-f4ef8f741ec2, which differs from the donor's UUID 3db9aaae-c037-4162-b0a8-9eec312df936. Manually drop the collection on this shard if it contains data from a previous incarnation of db.col
Sorry, I've changed the IPs and db and collection names due to security policy.
Here is the sharded collection status:
mongos> db.coll.getShardDistribution() Shard shard3 at shard3/IP:PORT data : 1.26GiB docs : 109259 chunks : 42 estimated data per chunk : 30.87MiB estimated docs per chunk : 2601 Shard shard1 at shard1/IP:PORT data : 1.18GiB docs : 102091 chunks : 36 estimated data per chunk : 33.61MiB estimated docs per chunk : 2835 Shard shard2 at shard2/IP:PORT data : 1.17GiB docs : 101178 chunks : 34 estimated data per chunk : 35.27MiB estimated docs per chunk : 2975 Totals data : 3.61GiB docs : 312528 chunks : 112 Shard shard3 contains 34.99% data, 34.95% docs in cluster, avg obj size on shard : 12KiB Shard shard1 contains 32.64% data, 32.66% docs in cluster, avg obj size on shard : 12KiB Shard shard2 contains 32.36% data, 32.37% docs in cluster, avg obj size on shard : 12KiB
The configuration is the following:
3 Shards with 5 nodes (PRIMARY + 4 Sec + 1 Arb)
Here is the Config log error related to the issue:
2019-10-20T02:00:04.541+0200 I SHARDING [Balancer] distributed lock 'db.coll' acquired for 'Migrating chunk(s) in collection db.coll', ts : 5daab1a4d74e87bf1a4ee992 2019-10-20T02:00:04.911+0200 I SHARDING [ShardRegistry] distributed lock with ts: '5daab1a4d74e87bf1a4ee992' and _id: 'db.coll' unlocked. 2019-10-20T02:00:04.922+0200 I SHARDING [Balancer] Balancer move db.coll: [{ productId: MinKey }, { productId: -9062096908397525382 }), from shard3, to shard2 failed :: caused by :: OperationFailed: Data transfer error: migrate failed: InvalidUUID: Cannot create collection db.coll because we already have an identically named collection with UUID 55ab81fa-7d21-4742-8d71-f4ef8f741ec2, which differs from the donor's UUID 3db9aaae-c037-4162-b0a8-9eec312df936. Manually drop the collection on this shard if it contains data from a previous incarnation of db.coll 2019-10-20T02:00:04.922+0200 I SHARDING [Balancer] about to log metadata event into actionlog: { _id: "host-2019-10-20T02:00:04.922+0200-5daba384d74e87bf1a699d86", server: "host", clientAddr: "", time: new Date(1571529604922), what: "balancer.round", ns: "", details: { executionTimeMillis: 465, errorOccured: false, candidateChunks: 1, chunksMoved: 0 } } 2019-10-20T02:00:14.962+0200 I SHARDING [Balancer] distributed lock 'db.coll' acquired for 'Migrating chunk(s) in collection db.coll', ts : 5daab1a4d74e87bf1a4ee992 2019-10-20T02:00:15.009+0200 I SHARDING [ShardRegistry] distributed lock with ts: '5daab1a4d74e87bf1a4ee992' and _id: 'db.coll' unlocked. 2019-10-20T02:00:15.012+0200 I SHARDING [Balancer] Balancer move db.coll: [{ productId: MinKey }, { productId: -9062096908397525382 }), from shard3, to shard2 failed :: caused by :: OperationFailed: Data transfer error: migrate failed: InvalidUUID: Cannot create collection db.coll because we already have an identically named collection with UUID 55ab81fa-7d21-4742-8d71-f4ef8f741ec2, which differs from the donor's UUID 3db9aaae-c037-4162-b0a8-9eec312df936. Manually drop the collection on this shard if it contains data from a previous incarnation of db.coll 2019-10-20T02:00:15.013+0200 I SHARDING [Balancer] about to log metadata event into actionlog: { _id: "host-2019-10-20T02:00:15.013+0200-5daba38fd74e87bf1a699e3d", server: "host", clientAddr: "", time: new Date(1571529615013), what: "balancer.round", ns: "", details: { executionTimeMillis: 82, errorOccured: false, candidateChunks: 1, chunksMoved: 0 } }
Please let me know if any additional info are require form my side.
Thank You