[SERVER-17893] Standalone server to Sharded cluster creates Jumbo chunks Created: 02/Apr/15  Updated: 06/Dec/22  Resolved: 09/Dec/19

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Jonathan Abrahams Assignee: [DO NOT USE] Backlog - Sharding Team
Resolution: Done Votes: 0
Labels: 32qa
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
Tested
Assigned Teams:
Sharding
Operating System: ALL
Participants:

 Description   

In testing a scenario where a large collection is sharded, the collection ends up with too many jumbo chunks, which prevent the chunks from being placed on the proper shard nodes (which are designated using addRangeTag).

Schema

  • {lname: <string>, fname: <string>, address: <string>, ssn: <string>, zip: <int>}
    • Note - doesn't matter if zip is int or string type
  • Indexes:
    • {ssn: 1}
    • {lname: 1, fname: 1}
    • {lname: 1, zip: 1}
    • {zip: 1}
  • Shard key: {zip:1}

Steps

  • On standalone (or replica set) node add 10 million documents using this basic query:

    function pad ( num, size ) {
      if (num.toString().length >= size) return num.toString();
      return ( Math.pow( 10, size ) + Math.floor(num) ).toString().substring( 1 );
    }
     
    function getRandomInt(digits) {
        return Math.floor(Math.random()*10000000)%Math.pow(10,digits);
    }
     
    function getSsn() {
        return pad(getRandomInt(3),3)+"-"+pad(getRandomInt(2),2)+"-"+pad(getRandomInt(4),4);
    }
     
    function getAddr () {
        return getRandomInt(3)+" "+
            streets[Math.floor(Math.random()*10000000)%streets.length]+" "+
            streetTypes[Math.floor(Math.random()*10000000)%streetTypes.length];
    }
     
    function getZip() {
        return pad(getRandomInt(5),5);
    }
     
    function getFname() {
        return firstNames[Math.floor(Math.random()*10000000)%firstNames.length];
    }
     
    function getLname() {
        return lastNames[Math.floor(Math.random()*10000000)%lastNames.length];
    }
     
    var cnt = db.users.count();
    x=Math.floor(cnt/1000);
    for (var i=x; i<50000; i++) {
        for (var j=0; j<1000; j++) {
            var doc = {ssn: getSsn(), lname: getLname(), fname: getFname(), addr: getAddr(), zip: getZip()};
            var bulk = db.users.initializeUnorderedBulkOp();
            bulk.insert(doc);
            var res = bulk.execute();
        }
        print((i+1)*j,tojson(doc));
    }
    

  • Add 2 more mongod processes and shard the replica set

    sh.addShard("<ip:port>");
    sh.addShard("<ip:port>");
    sh.addShard("<ip:port>");
    sh.enableSharding("test");
    sh.shardCollection("test.users", {zip: 1});
    db.adminCommand({ movePrimary : "test", to : "shard0000" });
    sh.status();
     
    // Shard Tags
    sh.addShardTag("rs0", "East");
    sh.addShardTag("rs1", "Central");
    sh.addShardTag("rs2", "West");
     
    sh.addTagRange("test.users", { zip: "00000" }, { zip: "39999" }, "East");
    sh.addTagRange("test.users", { zip: "40000" }, { zip: "79999" }, "Central");
    sh.addTagRange("test.users", { zip: "80000" }, { zip: "99999" }, "West");
    

As the chunks are auto-split, most will be marked as jumbo and will not be able to move to their tagged shard. Manual splitting still works on theses chunks.



 Comments   
Comment by Jonathan Abrahams [ 07/Apr/15 ]

I reran this scenario without the ShardTagRange as follows:

  • Use existing 3 node sharded cluster (the shards are tagged)
  • Use new db, i.e., test2
  • Populate collection users directly through mongod on rs0, with 10M documents
  • Enable sharding of the database
  • Move the primary to rs0 (for some reason it sets the primary to the last shard node, rs2)
  • Shard the collection
    At this point the chunks are created on rs0, with all but 1 being marked as jumbo
Comment by Jonathan Abrahams [ 07/Apr/15 ]

The problem is that migrating a replica set to a sharded cluster is creating jumbo chunks, from auto splitting, which can be manually split later. It seems that the mongos should auto split such that jumbo chunks are not created.

Comment by Jonathan Abrahams [ 02/Apr/15 ]

A manual split worked using the following commands:

// Split jumbo chunks
sh.stopBalancer();
var c = db.getSiblingDB("config").chunks.find({jumbo: true});
while (c.hasNext()) {
    var d = c.next();
    var min = d.min.zip;
    print("Split chunk", min);
    sh.splitFind("test.users", {zip: min});
}
sh.startBalancer();

So why would it fail with auto split?

Generated at Thu Feb 08 03:45:55 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.