Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-44143

moveChunk Issue (Mongo version 4.0.4)

    • Type: Icon: Bug Bug
    • Resolution: Incomplete
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Sharding
    • Labels:
      None
    • Fully Compatible
    • ALL

      Hello MongoDB Team.

      We faced with the issue related to moveChunk process.
      This is the error from one of the Shrad's log:

      2019-10-22T11:18:45.921+0200 I SHARDING [conn132] Starting chunk migration ns: db.col, [{ productId: MinKey }, { productId: -9062096908397525382 }), fromShard: shard3, toShard: shard2 with expected collection version epoch 5bb4b060aec28d86b2174007
      2019-10-22T11:18:45.924+0200 I SHARDING [conn132] about to log metadata event into changelog: { _id: "prulmcoembb09-2019-10-22T11:18:45.924+0200-5daec975680ef73c96f4be36", server: "prulmcoembb09", clientAddr: "10.217.1.252:50292", time: new Date(1571735925924), what: "moveChunk.start", ns: "db.col", details: { min: { productId: MinKey }, max: { productId: -9062096908397525382 }, from: "shard3", to: "shard2" } }
      2019-10-22T11:18:45.955+0200 I SHARDING [conn132] moveChunk data transfer progress: { waited: true, active: false, ns: "db.col", from: "shard3/IP:PORT", fromShardId: "shard3", min: { productId: MinKey }, max: { productId: -9062096908397525382 }, shardKeyPattern: { productId: "hashed" }, state: "fail", errmsg: "migrate failed: InvalidUUID: Cannot create collection db.col because we already have an identically named collection with UUID 55a...", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0, operationTime: Timestamp(1571735921, 1), $gleStats: { lastOpTime: Timestamp(0, 0), electionId: ObjectId('7fffffff000000000000001f') }, lastCommittedOpTime: Timestamp(1571735921, 1), $configServerState: { opTime: { ts: Timestamp(1571735925, 6), t: 18 } }, $clusterTime: { clusterTime: Timestamp(1571735925, 6), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } } } mem used: 0 documents remaining to clone: 2731
      2019-10-22T11:18:45.955+0200 I SHARDING [conn132] about to log metadata event into changelog: { _id: "prulmcoembb09-2019-10-22T11:18:45.955+0200-5daec975680ef73c96f4be5d", server: "prulmcoembb09", clientAddr: "10.217.1.252:50292", time: new Date(1571735925955), what: "moveChunk.error", ns: "db.col", details: { min: { productId: MinKey }, max: { productId: -9062096908397525382 }, from: "shard3", to: "shard2" } }
      2019-10-22T11:18:45.961+0200 W SHARDING [conn132] Chunk move failed :: caused by :: OperationFailed: Data transfer error: migrate failed: InvalidUUID: Cannot create collection db.col because we already have an identically named collection with UUID 55ab81fa-7d21-4742-8d71-f4ef8f741ec2, which differs from the donor's UUID 3db9aaae-c037-4162-b0a8-9eec312df936. Manually drop the collection on this shard if it contains data from a previous incarnation of db.col

       

      Sorry, I've changed the IPs and db and collection names due to security policy.

      Here is the sharded collection status:

       

       

      mongos> db.coll.getShardDistribution()
      Shard shard3 at shard3/IP:PORT
       data : 1.26GiB docs : 109259 chunks : 42
       estimated data per chunk : 30.87MiB
       estimated docs per chunk : 2601
      Shard shard1 at shard1/IP:PORT
       data : 1.18GiB docs : 102091 chunks : 36
       estimated data per chunk : 33.61MiB
       estimated docs per chunk : 2835
      Shard shard2 at shard2/IP:PORT
       data : 1.17GiB docs : 101178 chunks : 34
       estimated data per chunk : 35.27MiB
       estimated docs per chunk : 2975
      Totals
       data : 3.61GiB docs : 312528 chunks : 112
       Shard shard3 contains 34.99% data, 34.95% docs in cluster, avg obj size on shard : 12KiB
       Shard shard1 contains 32.64% data, 32.66% docs in cluster, avg obj size on shard : 12KiB
       Shard shard2 contains 32.36% data, 32.37% docs in cluster, avg obj size on shard : 12KiB
      

       The configuration is the following:

      3 Shards with 5 nodes (PRIMARY + 4 Sec + 1 Arb)

      Here is the Config log error related to the issue:

      2019-10-20T02:00:04.541+0200 I SHARDING [Balancer] distributed lock 'db.coll' acquired for 'Migrating chunk(s) in collection db.coll', ts : 5daab1a4d74e87bf1a4ee992
      2019-10-20T02:00:04.911+0200 I SHARDING [ShardRegistry] distributed lock with ts: '5daab1a4d74e87bf1a4ee992' and _id: 'db.coll' unlocked.
      2019-10-20T02:00:04.922+0200 I SHARDING [Balancer] Balancer move db.coll: [{ productId: MinKey }, { productId: -9062096908397525382 }), from shard3, to shard2 failed :: caused by :: OperationFailed: Data transfer error: migrate failed: InvalidUUID: Cannot create collection db.coll because we already have an identically named collection with UUID 55ab81fa-7d21-4742-8d71-f4ef8f741ec2, which differs from the donor's UUID 3db9aaae-c037-4162-b0a8-9eec312df936. Manually drop the collection on this shard if it contains data from a previous incarnation of db.coll
      2019-10-20T02:00:04.922+0200 I SHARDING [Balancer] about to log metadata event into actionlog: { _id: "host-2019-10-20T02:00:04.922+0200-5daba384d74e87bf1a699d86", server: "host", clientAddr: "", time: new Date(1571529604922), what: "balancer.round", ns: "", details: { executionTimeMillis: 465, errorOccured: false, candidateChunks: 1, chunksMoved: 0 } }
      2019-10-20T02:00:14.962+0200 I SHARDING [Balancer] distributed lock 'db.coll' acquired for 'Migrating chunk(s) in collection db.coll', ts : 5daab1a4d74e87bf1a4ee992
      2019-10-20T02:00:15.009+0200 I SHARDING [ShardRegistry] distributed lock with ts: '5daab1a4d74e87bf1a4ee992' and _id: 'db.coll' unlocked.
      2019-10-20T02:00:15.012+0200 I SHARDING [Balancer] Balancer move db.coll: [{ productId: MinKey }, { productId: -9062096908397525382 }), from shard3, to shard2 failed :: caused by :: OperationFailed: Data transfer error: migrate failed: InvalidUUID: Cannot create collection db.coll because we already have an identically named collection with UUID 55ab81fa-7d21-4742-8d71-f4ef8f741ec2, which differs from the donor's UUID 3db9aaae-c037-4162-b0a8-9eec312df936. Manually drop the collection on this shard if it contains data from a previous incarnation of db.coll
      2019-10-20T02:00:15.013+0200 I SHARDING [Balancer] about to log metadata event into actionlog: { _id: "host-2019-10-20T02:00:15.013+0200-5daba38fd74e87bf1a699e3d", server: "host", clientAddr: "", time: new Date(1571529615013), what: "balancer.round", ns: "", details: { executionTimeMillis: 82, errorOccured: false, candidateChunks: 1, chunksMoved: 0 } }

      Please let me know if any additional info are require form my side.

      Thank You

            Assignee:
            dmitry.agranat@mongodb.com Dmitry Agranat
            Reporter:
            genacvali91 Gennadiy
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: