Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-13491

chunks problem at initial splits for empty hashed collection

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major - P3
    • Resolution: Cannot Reproduce
    • 2.4.9, 2.4.10
    • None
    • Sharding
    • ALL
    • Hide

      see SERVER-12638 script

      Show
      see SERVER-12638 script

    Description

      Not all chunks are moved or created correct! Config unusable for some collections!

      Test Shard: 4 single shards(no rs), 3 config, 1mongos

      Create 10 threads:
      every thread create 1 Schema and 1 collection with enable sharding.
      always hashed shardKey on _id and no numInitialChunks

      case1 (with deaktivated balancer)
      not all chunks are moved or created correct!

             {  "_id" : "ci_400000000000005",  "partitioned" : true,  "primary" : "shard_001" }
                     ci_400000000000005.customer_informations
                             shard key: { "_id" : "hashed" }
                             chunks:
                                     shard_002       1
                                     shard_003       1
                                     shard_004       1
                                     shard_001       1
                             { "_id" : { "$minKey" : 1 } } -->> { "_id" : NumberLong("-4611686018427387900") } on : shard_002 Timestamp(2, 0)
                             { "_id" : NumberLong("-4611686018427387900") } -->> { "_id" : NumberLong(0) } on : shard_003 Timestamp(3, 0)
                             { "_id" : NumberLong(0) } -->> { "_id" : NumberLong("4611686018427387900") } on : shard_004 Timestamp(4, 0)
                             { "_id" : NumberLong("4611686018427387900") } -->> { "_id" : { "$maxKey" : 1 } } on : shard_001 Timestamp(4, 1)
       
       
           {  "_id" : "ci_400000000000019",  "partitioned" : true,  "primary" : "shard_004" }
                   ci_400000000000019.customer_informations
                           shard key: { "_id" : "hashed" }
                           chunks:
                                   shard_004       8
                           { "_id" : { "$minKey" : 1 } } -->> { "_id" : NumberLong("-6917529027641081850") } on : shard_004 Timestamp(1, 4)
                           { "_id" : NumberLong("-6917529027641081850") } -->> { "_id" : NumberLong("-4611686018427387900") } on : shard_004 Timestamp(1, 5)
                           { "_id" : NumberLong("-4611686018427387900") } -->> { "_id" : NumberLong("-2305843009213693950") } on : shard_004 Timestamp(1, 6)
                           { "_id" : NumberLong("-2305843009213693950") } -->> { "_id" : NumberLong(0) } on : shard_004 Timestamp(1, 7)
                           { "_id" : NumberLong(0) } -->> { "_id" : NumberLong("2305843009213693950") } on : shard_004 Timestamp(1, 8)
                           { "_id" : NumberLong("2305843009213693950") } -->> { "_id" : NumberLong("4611686018427387900") } on : shard_004 Timestamp(1, 9)
                           { "_id" : NumberLong("4611686018427387900") } -->> { "_id" : NumberLong("6917529027641081850") } on : shard_004 Timestamp(1, 10)
                           { "_id" : NumberLong("6917529027641081850") } -->> { "_id" : { "$maxKey" : 1 } } on : shard_004 Timestamp(1, 11)
       
       
           {  "_id" : "ci_400000000000025",  "partitioned" : true,  "primary" : "shard_003" }
                   ci_400000000000025.customer_informations
                           shard key: { "_id" : "hashed" }
                           chunks:
                                   shard_001       2
                                   shard_002       2
                                   shard_004       2
                                   shard_003       2
                           { "_id" : { "$minKey" : 1 } } -->> { "_id" : NumberLong("-6917529027641081850") } on : shard_001 Timestamp(2, 0)
                           { "_id" : NumberLong("-6917529027641081850") } -->> { "_id" : NumberLong("-4611686018427387900") } on : shard_002 Timestamp(3, 0)
                           { "_id" : NumberLong("-4611686018427387900") } -->> { "_id" : NumberLong("-2305843009213693950") } on : shard_004 Timestamp(4, 0)
                           { "_id" : NumberLong("-2305843009213693950") } -->> { "_id" : NumberLong(0) } on : shard_001 Timestamp(5, 0)
                           { "_id" : NumberLong(0) } -->> { "_id" : NumberLong("2305843009213693950") } on : shard_002 Timestamp(6, 0)
                           { "_id" : NumberLong("2305843009213693950") } -->> { "_id" : NumberLong("4611686018427387900") } on : shard_004 Timestamp(7, 0)
                           { "_id" : NumberLong("4611686018427387900") } -->> { "_id" : NumberLong("6917529027641081850") } on : shard_003 Timestamp(7, 1)
                           { "_id" : NumberLong("6917529027641081850") } -->> { "_id" : { "$maxKey" : 1 } } on : shard_003 Timestamp(1, 11)
       
             {  "_id" : "ci_400000000000006",  "partitioned" : true,  "primary" : "shard_002" }
                     ci_400000000000006.customer_informations
                             shard key: { "_id" : "hashed" }
                             chunks:
                                     shard_001       4
                             { "_id" : { "$minKey" : 1 } } -->> { "_id" : NumberLong("-4611686018427387900") } on : shard_001 Timestamp(1, 0)
                             { "_id" : NumberLong("-4611686018427387900") } -->> { "_id" : NumberLong(0) } on : shard_001 Timestamp(1, 1)
                             { "_id" : NumberLong(0) } -->> { "_id" : NumberLong("4611686018427387900") } on : shard_001 Timestamp(1, 2)
                             { "_id" : NumberLong("4611686018427387900") } -->> { "_id" : { "$maxKey" : 1 } } on : shard_001 Timestamp(1, 3)

      case 2 (with active balancer)
      not all chunks are moved or created correct!

        {  "_id" : "ci_400000000000018",  "partitioned" : true,  "primary" : "shard_003" }
                ci_400000000000018.customer_informations
                        shard key: { "_id" : "hashed" }
                        chunks:
                                shard_003       4
                        { "_id" : { "$minKey" : 1 } } -->> { "_id" : NumberLong("-4611686018427387900") } on : shard_003 Timestamp(1, 0)
                        { "_id" : NumberLong("-4611686018427387900") } -->> { "_id" : NumberLong(0) } on : shard_003 Timestamp(1, 1)
                        { "_id" : NumberLong(0) } -->> { "_id" : NumberLong("4611686018427387900") } on : shard_003 Timestamp(1, 2)
                        { "_id" : NumberLong("4611686018427387900") } -->> { "_id" : { "$maxKey" : 1 } } on : shard_003 Timestamp(1, 3)
      				  
      Fri Apr 04 19:52:29.467 [Balancer] moving chunk ns: ci_400000000000018.customer_informations moving ( ns:ci_400000000000018.customer_informationsshard: shard_003:127.0.0.1:20317lastmod: 1|0||000000000000000000000000min: { _id: MinKey }max: { _id: -4611686018427387900 }) shard_003:127.0.0.1:20317 -> shard_001:127.0.0.1:20117
      Fri Apr 04 19:52:29.552 [Balancer] moveChunk result: { who: { _id: "ci_400000000000018.customer_informations", process: "WT-INT34:20317:1396633596:41", state: 1, ts: ObjectId('533ef0092b8745cddb99e33e'), when: new Date(1396633609505), who: "WT-INT34:20317:1396633596:41:conn17:41", why: "migrate-{ _id: MinKey }" }, ok: 0.0, errmsg: "the collection metadata could not be locked with lock migrate-{ _id: MinKey }" }
      Fri Apr 04 19:52:29.552 [Balancer] balancer move failed: { who: { _id: "ci_400000000000018.customer_informations", process: "WT-INT34:20317:1396633596:41", state: 1, ts: ObjectId('533ef0092b8745cddb99e33e'), when: new Date(1396633609505), who: "WT-INT34:20317:1396633596:41:conn17:41", why: "migrate-{ _id: MinKey }" }, ok: 0.0, errmsg: "the collection metadata could not be locked with lock migrate-{ _id: MinKey }" } from: shard_003 to: shard_001 chunk:  min: { _id: MinKey } max: { _id: -4611686018427387900 }

      See
      SERVER-12515 and SERVER-12638

      Attachments

        1. 20140407_CorruptConfigsWithManyChunks_correctSorted.txt
          160 kB
        2. 20140407_CorruptConfigsWithManyChunks.txt
          160 kB
        3. 20140407_createCollectionsBalancerDisabled.rar
          47 kB
        4. 20140407_createCollectionsBalancerEnabled.rar
          53 kB
        5. 20140409_curruptCreatedCollections.rar
          14 kB
        6. 20140430_1630_mongos1.log
          148 kB

        Activity

          People

            thomas.rueckstiess@mongodb.com Thomas Rueckstiess
            Björn Bullerdieck Björn Bullerdieck [X]
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: