Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-26688

Chunks consisting of a single shard key value do not mark Jumbo when move/split fails

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Duplicate
    • Affects Version/s: 3.2.8
    • Fix Version/s: None
    • Component/s: Sharding
    • Labels:
      None
    • Operating System:
      ALL
    • Steps To Reproduce:
      Hide

      use test;
      db.c.createIndex( { uid: 1} )
      use config;
      sh.enableSharding("test")
      sh.shardCollection("test.c", { uid: 1 } )
      sh.stopBalancer();
      db.adminCommand({split : "test.c", middle : {uid : 10} })
      db.adminCommand({split : "test.c", middle : {uid : 100} })
      db.adminCommand({split : "test.c", middle : {uid : 1000} })
      db.settings.find();
      // we want:
      //     { "_id" : "chunksize", "value" : NumberLong(64) }
      //     { "_id" : "balancer", "stopped" : true }
      use test;
      // doc ~270 B size, should get us a ~72 MB chunk with 251K docs - leave 8 B added by uid
      var doc = {
       "a": ".23456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.1234567890123456789012345678901234567890123456789012345678901"
      };
      for (var ix=0; ix < 251000; ix++) {
        doc.uid = 99;
        // if ( ix % 2 == 0 ) { doc.uid = 100; }  // should split ok into 99 and 100 -only chunks when chunkSize > maxChunkSize/2 (avoid SERVER-19919)
        db.c.insert( doc );
        if ( ix % 10000 == 0 ) { print(ix); }
      }
      print(ix);
      db.c.stats();
      sh.status(true);
      

      Note the balancer is off and the Jumbo flags are absent, as initially expected:

      mongos> sh.status(true)
      --- Sharding Status --- 
        sharding version: {
      	"_id" : 1,
      	"minCompatibleVersion" : 5,
      	"currentVersion" : 6,
      	"clusterId" : ObjectId("57fd594ff35daed3247bea8d")
      }
        shards:
      	{  "_id" : "sh1",  "host" : "sh1/localhost:27018,localhost:27019" }
      	{  "_id" : "sh2",  "host" : "sh2/localhost:27020,localhost:27021" }
        active mongoses:
      	{  "_id" : "Scotts-MacBook-Pro.local:27017",  "ping" : ISODate("2016-10-11T21:41:26.026Z"),  "up" : NumberLong(822),  "waiting" : true,  "mongoVersion" : "3.2.8" }
        balancer:
      	Currently enabled:  no
      	Currently running:  no
      	Failed balancer rounds in last 5 attempts:  0
      	Migration Results for the last 24 hours: 
      		No recent migrations
        databases:
      	{  "_id" : "test",  "primary" : "sh2",  "partitioned" : true }
      		test.c
      			shard key: { "uid" : 1 }
      			unique: false
      			balancing: true
      			chunks:
      				sh2	4
      			{ "uid" : { "$minKey" : 1 } } -->> { "uid" : 10 } on : sh2 Timestamp(1, 1) 
      			{ "uid" : 10 } -->> { "uid" : 100 } on : sh2 Timestamp(1, 3) 
      			{ "uid" : 100 } -->> { "uid" : 1000 } on : sh2 Timestamp(1, 5) 
      			{ "uid" : 1000 } -->> { "uid" : { "$maxKey" : 1 } } on : sh2 Timestamp(1, 6) 
      

      Finally, start the balancer,

      mongos> sh.startBalancer();
      

      and note the server log entries and no-Jumbo flag result:

      2016-10-11T14:42:27.572-0700 I SHARDING [Balancer] moving chunk ns: test.c moving ( ns: test.c, shard: sh2, lastmod: 2|1||57fd5ae3f35daed3247beaa7, min: { uid: 10.0 }, max: { uid: 100.0 }) sh2 -> sh1
      2016-10-11T14:42:27.799-0700 I SHARDING [Balancer] moveChunk result: { chunkTooBig: true, estimatedChunkSize: 76304000, ok: 0.0, errmsg: "chunk too big to move" }
      2016-10-11T14:42:27.801-0700 I SHARDING [Balancer] balancer move failed: { chunkTooBig: true, estimatedChunkSize: 76304000, ok: 0.0, errmsg: "chunk too big to move" } from: sh2 to: sh1 chunk:  min: { uid: 10.0 } max: { uid: 100.0 }
      2016-10-11T14:42:27.801-0700 I SHARDING [Balancer] performing a split because migrate failed for size reasons
      2016-10-11T14:42:28.085-0700 I SHARDING [Balancer] split results: CannotSplit: chunk not full enough to trigger auto-split
      2016-10-11T14:42:28.086-0700 I SHARDING [Balancer] marking chunk as jumbo: ns: test.c, shard: sh2, lastmod: 2|1||57fd5ae3f35daed3247beaa7, min: { uid: 10.0 }, max: { uid: 100.0 }
      

      Inspecting the related source code for these messages ...

      Suggests the state of txn is deficient to permit setting the chunk's Jumbo flag. The resulting shard status is absent the Jumbo flag that should have been set:

      mongos> sh.status(true)
      --- Sharding Status --- 
        sharding version: {
      	"_id" : 1,
      	"minCompatibleVersion" : 5,
      	"currentVersion" : 6,
      	"clusterId" : ObjectId("57fd594ff35daed3247bea8d")
      }
        shards:
      	{  "_id" : "sh1",  "host" : "sh1/localhost:27018,localhost:27019" }
      	{  "_id" : "sh2",  "host" : "sh2/localhost:27020,localhost:27021" }
        active mongoses:
      	{  "_id" : "Scotts-MacBook-Pro.local:27017",  "ping" : ISODate("2016-10-11T21:55:54.530Z"),  "up" : NumberLong(1690),  "waiting" : true,  "mongoVersion" : "3.2.8" }
        balancer:
      	Currently enabled:  yes
      	Currently running:  no
      	Failed balancer rounds in last 5 attempts:  0
      	Migration Results for the last 24 hours: 
      		2 : Success
      		1 : Failed with error 'chunk too big to move', from sh2 to sh1
        databases:
      	{  "_id" : "test",  "primary" : "sh2",  "partitioned" : true }
      		test.c
      			shard key: { "uid" : 1 }
      			unique: false
      			balancing: true
      			chunks:
      				sh1	2
      				sh2	2
      			{ "uid" : { "$minKey" : 1 } } -->> { "uid" : 10 } on : sh2 Timestamp(7, 0) 
      			{ "uid" : 10 } -->> { "uid" : 100 } on : sh2 Timestamp(6, 1) 
      			{ "uid" : 100 } -->> { "uid" : 1000 } on : sh1 Timestamp(7, 1) 
      			{ "uid" : 1000 } -->> { "uid" : { "$maxKey" : 1 } } on : sh1 Timestamp(4, 0) 
      

      Repeating with collection 'C' inserting TWO shard key values, uid = 99, 100 – splits normally:

      use test;
      db.C.createIndex( { uid: 1} )
      use config;
      sh.enableSharding("test")
      sh.shardCollection("test.C", { uid: 1 } )
      sh.stopBalancer();
      db.adminCommand({split : "test.C", middle : {uid : 10} })
      db.adminCommand({split : "test.C", middle : {uid : 100} })
      db.adminCommand({split : "test.C", middle : {uid : 1000} })
      db.settings.find();
      // we want:
      //     { "_id" : "chunksize", "value" : NumberLong(64) }
      //     { "_id" : "balancer", "stopped" : true }
      use test;
      // doc ~270 B size, should get us a ~72 MB chunk with 251K docs - leave 8 B added by uid
      var doc = {
       "a": ".23456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.1234567890123456789012345678901234567890123456789012345678901"
      };
      for (var ix=0; ix < 251000; ix++) {
        doc.uid = 99;
        if ( ix % 2 == 0 ) { doc.uid = 100; }  // should split ok into 99 and 100 -only chunks when chunkSize > maxChunkSize/2 (avoid SERVER-19919)
        db.C.insert( doc );
        if ( ix % 10000 == 0 ) { print(ix); }
      }
      print(ix);
      db.C.stats();
      sh.status(true);
      mongos> sh.status(true);
      

      Show
      use test; db.c.createIndex( { uid: 1} ) use config; sh.enableSharding("test") sh.shardCollection("test.c", { uid: 1 } ) sh.stopBalancer(); db.adminCommand({split : "test.c", middle : {uid : 10} }) db.adminCommand({split : "test.c", middle : {uid : 100} }) db.adminCommand({split : "test.c", middle : {uid : 1000} }) db.settings.find(); // we want: // { "_id" : "chunksize", "value" : NumberLong(64) } // { "_id" : "balancer", "stopped" : true } use test; // doc ~270 B size, should get us a ~72 MB chunk with 251K docs - leave 8 B added by uid var doc = { "a": ".23456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.1234567890123456789012345678901234567890123456789012345678901" }; for (var ix=0; ix < 251000; ix++) { doc.uid = 99; // if ( ix % 2 == 0 ) { doc.uid = 100; } // should split ok into 99 and 100 -only chunks when chunkSize > maxChunkSize/2 (avoid SERVER-19919) db.c.insert( doc ); if ( ix % 10000 == 0 ) { print(ix); } } print(ix); db.c.stats(); sh.status(true); Note the balancer is off and the Jumbo flags are absent, as initially expected: mongos> sh.status(true) --- Sharding Status --- sharding version: { "_id" : 1, "minCompatibleVersion" : 5, "currentVersion" : 6, "clusterId" : ObjectId("57fd594ff35daed3247bea8d") } shards: { "_id" : "sh1", "host" : "sh1/localhost:27018,localhost:27019" } { "_id" : "sh2", "host" : "sh2/localhost:27020,localhost:27021" } active mongoses: { "_id" : "Scotts-MacBook-Pro.local:27017", "ping" : ISODate("2016-10-11T21:41:26.026Z"), "up" : NumberLong(822), "waiting" : true, "mongoVersion" : "3.2.8" } balancer: Currently enabled: no Currently running: no Failed balancer rounds in last 5 attempts: 0 Migration Results for the last 24 hours: No recent migrations databases: { "_id" : "test", "primary" : "sh2", "partitioned" : true } test.c shard key: { "uid" : 1 } unique: false balancing: true chunks: sh2 4 { "uid" : { "$minKey" : 1 } } -->> { "uid" : 10 } on : sh2 Timestamp(1, 1) { "uid" : 10 } -->> { "uid" : 100 } on : sh2 Timestamp(1, 3) { "uid" : 100 } -->> { "uid" : 1000 } on : sh2 Timestamp(1, 5) { "uid" : 1000 } -->> { "uid" : { "$maxKey" : 1 } } on : sh2 Timestamp(1, 6) Finally, start the balancer, mongos> sh.startBalancer(); and note the server log entries and no-Jumbo flag result: 2016-10-11T14:42:27.572-0700 I SHARDING [Balancer] moving chunk ns: test.c moving ( ns: test.c, shard: sh2, lastmod: 2|1||57fd5ae3f35daed3247beaa7, min: { uid: 10.0 }, max: { uid: 100.0 }) sh2 -> sh1 2016-10-11T14:42:27.799-0700 I SHARDING [Balancer] moveChunk result: { chunkTooBig: true, estimatedChunkSize: 76304000, ok: 0.0, errmsg: "chunk too big to move" } 2016-10-11T14:42:27.801-0700 I SHARDING [Balancer] balancer move failed: { chunkTooBig: true, estimatedChunkSize: 76304000, ok: 0.0, errmsg: "chunk too big to move" } from: sh2 to: sh1 chunk: min: { uid: 10.0 } max: { uid: 100.0 } 2016-10-11T14:42:27.801-0700 I SHARDING [Balancer] performing a split because migrate failed for size reasons 2016-10-11T14:42:28.085-0700 I SHARDING [Balancer] split results: CannotSplit: chunk not full enough to trigger auto-split 2016-10-11T14:42:28.086-0700 I SHARDING [Balancer] marking chunk as jumbo: ns: test.c, shard: sh2, lastmod: 2|1||57fd5ae3f35daed3247beaa7, min: { uid: 10.0 }, max: { uid: 100.0 } Inspecting the related source code for these messages ... "performing a split because migrate failed for size reasons" calls _splitOrMarkJumbo() "CannotSplit: chunk not full enough to trigger auto-split" "marking chunk as jumbo" calls chunk->markAsJumbo(txn) Suggests the state of txn is deficient to permit setting the chunk's Jumbo flag. The resulting shard status is absent the Jumbo flag that should have been set: mongos> sh.status(true) --- Sharding Status --- sharding version: { "_id" : 1, "minCompatibleVersion" : 5, "currentVersion" : 6, "clusterId" : ObjectId("57fd594ff35daed3247bea8d") } shards: { "_id" : "sh1", "host" : "sh1/localhost:27018,localhost:27019" } { "_id" : "sh2", "host" : "sh2/localhost:27020,localhost:27021" } active mongoses: { "_id" : "Scotts-MacBook-Pro.local:27017", "ping" : ISODate("2016-10-11T21:55:54.530Z"), "up" : NumberLong(1690), "waiting" : true, "mongoVersion" : "3.2.8" } balancer: Currently enabled: yes Currently running: no Failed balancer rounds in last 5 attempts: 0 Migration Results for the last 24 hours: 2 : Success 1 : Failed with error 'chunk too big to move', from sh2 to sh1 databases: { "_id" : "test", "primary" : "sh2", "partitioned" : true } test.c shard key: { "uid" : 1 } unique: false balancing: true chunks: sh1 2 sh2 2 { "uid" : { "$minKey" : 1 } } -->> { "uid" : 10 } on : sh2 Timestamp(7, 0) { "uid" : 10 } -->> { "uid" : 100 } on : sh2 Timestamp(6, 1) { "uid" : 100 } -->> { "uid" : 1000 } on : sh1 Timestamp(7, 1) { "uid" : 1000 } -->> { "uid" : { "$maxKey" : 1 } } on : sh1 Timestamp(4, 0) Repeating with collection 'C' inserting TWO shard key values, uid = 99, 100 – splits normally: use test; db.C.createIndex( { uid: 1} ) use config; sh.enableSharding("test") sh.shardCollection("test.C", { uid: 1 } ) sh.stopBalancer(); db.adminCommand({split : "test.C", middle : {uid : 10} }) db.adminCommand({split : "test.C", middle : {uid : 100} }) db.adminCommand({split : "test.C", middle : {uid : 1000} }) db.settings.find(); // we want: // { "_id" : "chunksize", "value" : NumberLong(64) } // { "_id" : "balancer", "stopped" : true } use test; // doc ~270 B size, should get us a ~72 MB chunk with 251K docs - leave 8 B added by uid var doc = { "a": ".23456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.1234567890123456789012345678901234567890123456789012345678901" }; for (var ix=0; ix < 251000; ix++) { doc.uid = 99; if ( ix % 2 == 0 ) { doc.uid = 100; } // should split ok into 99 and 100 -only chunks when chunkSize > maxChunkSize/2 (avoid SERVER-19919) db.C.insert( doc ); if ( ix % 10000 == 0 ) { print(ix); } } print(ix); db.C.stats(); sh.status(true); mongos> sh.status(true);

      Description

      Inserting 250001+ small documents (~270B) having a highly-skewed corner case of a single shard key value results in a chunk that

      • meets the size-by-count criteria for a split (has > 250K docs), and
      • meets the size-by-bytes criteria for an auto-split (has > maxChunkSize bytes), but
      • apparently because of 0 computed split points in a single shard key value, it should be marked Jumbo, but the Jumbo flag does not get set by the balancer, even though the server log says it did so.

      If the shard is drained, however, using the removeShard and movePrimary commands, the balancer correctly marks the same chunk as Jumbo, as expected.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned
              Reporter:
              scott.kurowski Scott Kurowski
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: