[SERVER-7960] Chunk size different on shards of same MongoDB cluster Created: 17/Dec/12  Updated: 08/Mar/13  Resolved: 23/Dec/12

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 2.2.1
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Santosh Kumar L Assignee: Stennie Steneker (Inactive)
Resolution: Cannot Reproduce Votes: 0
Labels: Router, chunkSize, chunks, mongos, sharding
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux


Operating System: Linux
Steps To Reproduce:

Set up a 6 node shard and start the router without explicitly specifying the values for chunkSize and oplogSize parameters.
Load the data into the cluster using YCSB clients.
Check the distribution of data once the data loading is done.
For this log in to any of the routers and navigate to the database and then issue the following command.
db.<collection>.getShardDistribution()

Participants:

 Description   

We have setup a 6 Shard MongoDB cluster with a replication factor of 3.
When starting the router process, default chunk size and oplog size was chosen by not specifying the values for these explicitly.

Shard3 has a chunk size of 161 MB while the rest have 60-90 MB per chunk.
All shards are similar type of instances on Amazon EC2 environment.
What we have noticed using db.<collection>.getShardDistribution() command is as follows:

Shard shard1 at shard1/<ips of shard1>
data : 38.8Gb docs : 43049426 chunks : 621
estimated data per chunk : 63.99Mb
estimated docs per chunk : 69322

Shard shard2 at shard2/<ips of shard2>
data : 40.24Gb docs : 44644092 chunks : 620
estimated data per chunk : 66.47Mb
estimated docs per chunk : 72006

Shard shard3 at shard3/<ips of shard3>
data : 102.65Gb docs : 113874252 chunks : 649
estimated data per chunk : 161.97Mb
estimated docs per chunk : 175461

Shard shard4 at shard4/<ips of shard4>
data : 54.51Gb docs : 60472368 chunks : 620
estimated data per chunk : 90.04Mb
estimated docs per chunk : 97536

Shard shard5 at shard5/<ips of shard5>
data : 50.48Gb docs : 56005174 chunks : 620
estimated data per chunk : 83.38Mb
estimated docs per chunk : 90330

Shard shard6 at shard6/<ips of shard6>
data : 46.32Gb docs : 51388397 chunks : 620
estimated data per chunk : 76.51Mb
estimated docs per chunk : 82884

Totals
data : 333.05Gb docs : 369433709 chunks : 3750
Shard shard1 contains 11.65% data, 11.65% docs in cluster, avg obj size on shard : 967b
Shard shard2 contains 12.08% data, 12.08% docs in cluster, avg obj size on shard : 967b
Shard shard3 contains 30.82% data, 30.82% docs in cluster, avg obj size on shard : 967b
Shard shard4 contains 16.36% data, 16.36% docs in cluster, avg obj size on shard : 967b
Shard shard5 contains 15.15% data, 15.15% docs in cluster, avg obj size on shard : 967b
Shard shard6 contains 13.91% data, 13.91% docs in cluster, avg obj size on shard : 967b



 Comments   
Comment by Stennie Steneker (Inactive) [ 23/Dec/12 ]

Hi Santosh,

Without the data available, we cannot help troubleshoot this. My suspicion is that the shard key chosen did not provide sufficient uniqueness and so some "jumbo chunks" were created. Jumbo chunks cannot be split, so will continue to grow; they will also not be migrated by the balancer.

The Uniform distribution for YCSB also only appears to be in relation to generating load, not uniqueness of keys for sharding:

Uniform: Choose an item uniformly at random. For example, when choosing a record, all records in the database are equally likely to be chosen.

If you re-run this benchmark in future and still see the same issue, I would suggest starting a discussion on the mongodb-users group: http://groups.google.com/group/mongodb-user

Others testing YCSB with sharding will be able to share their feedback, and if there turns out to be a server issue we would then raise a Jira for it.

Cheers,
Stephen

Comment by Santosh Kumar L [ 21/Dec/12 ]

We have terminated the cluster so I cannot provide you the output of db.chunks.find(

{jumbo:true}

) command.
All the records inserted in the database are of 1 KB size each and "_id" field is the shard key being used. With uniform distribution of YCSB, all the records will have unique value of "_id" value.
So I want to understand why mongodb is unable to have chunks of 64MB on all the shards???

Comment by Stennie Steneker (Inactive) [ 21/Dec/12 ]

Hi Santosh,

It looks like your chunks are balanced except on shard3. I expect the size of the chunks is related to your choice of shard key. Perhaps you have some "jumbo chunks" which cannot be split or migrated on shard3.

Can you include the output of:

use config;
db.chunks.find({jumbo:true})

Thanks,
Stephen

Comment by Santosh Kumar L [ 21/Dec/12 ]

PFB sh.status() output.

mongos> sh.status()
— Sharding Status —
sharding version:

{ "_id" : 1, "version" : 3 }

shards:

{ "_id" : "shard1", "host" : "<ip:port>" } { "_id" : "shard2", "host" : "<ip:port>" } { "_id" : "shard3", "host" : "<ip:port>" } { "_id" : "shard4", "host" : "<ip:port>" } { "_id" : "shard5", "host" : "<ip:port>" } { "_id" : "shard6", "host" : "<ip:port>" }

databases:

{ "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "test", "partitioned" : true, "primary" : "shard3" }

test.usertable chunks:
shard1 621
shard2 620
shard3 649
shard4 620
shard5 620
shard6 620
too many chunks to print, use verbose if you want to force print

Comment by Santosh Kumar L [ 21/Dec/12 ]

We are using YCSB jar for inserting data into MongoDB cluster.
YCSB inserts serially into the DB. So these are normal inserts. Not bulk inserts.
YCSB jar file requires the following data as input parameters.

1.mongos router ip:port
2.database name in MongoDB cluster
3.No of records to be inserted
4.Distribution type to be used for inserting data.

regards,
Santosh

Comment by Eliot Horowitz (Inactive) [ 21/Dec/12 ]

Can you send the exact way you are running YCSB?

Comment by Stennie Steneker (Inactive) [ 21/Dec/12 ]

Hi Santosh,

Can you post the info from sh.status()?

Thanks,
Stephen

Comment by Santosh Kumar L [ 21/Dec/12 ]

I'm inserting data using YCSB framework.
These are inserts as I'm trying to load the data into the MongoDB cluster.

Comment by Eliot Horowitz (Inactive) [ 21/Dec/12 ]

How are you inserting data?
insert? bulk insert? upsert? etc...?

Generated at Thu Feb 08 03:16:09 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.