[SERVER-7960] Chunk size different on shards of same MongoDB cluster Created: 17/Dec/12 Updated: 08/Mar/13 Resolved: 23/Dec/12 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 2.2.1 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Santosh Kumar L | Assignee: | Stennie Steneker (Inactive) |
| Resolution: | Cannot Reproduce | Votes: | 0 |
| Labels: | Router, chunkSize, chunks, mongos, sharding | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Linux |
||
| Operating System: | Linux |
| Steps To Reproduce: | Set up a 6 node shard and start the router without explicitly specifying the values for chunkSize and oplogSize parameters. |
| Participants: |
| Description |
|
We have setup a 6 Shard MongoDB cluster with a replication factor of 3. Shard3 has a chunk size of 161 MB while the rest have 60-90 MB per chunk. Shard shard1 at shard1/<ips of shard1> Shard shard2 at shard2/<ips of shard2> Shard shard3 at shard3/<ips of shard3> Shard shard4 at shard4/<ips of shard4> Shard shard5 at shard5/<ips of shard5> Shard shard6 at shard6/<ips of shard6> Totals |
| Comments |
| Comment by Stennie Steneker (Inactive) [ 23/Dec/12 ] | ||
|
Hi Santosh, Without the data available, we cannot help troubleshoot this. My suspicion is that the shard key chosen did not provide sufficient uniqueness and so some "jumbo chunks" were created. Jumbo chunks cannot be split, so will continue to grow; they will also not be migrated by the balancer. The Uniform distribution for YCSB also only appears to be in relation to generating load, not uniqueness of keys for sharding:
If you re-run this benchmark in future and still see the same issue, I would suggest starting a discussion on the mongodb-users group: http://groups.google.com/group/mongodb-user Others testing YCSB with sharding will be able to share their feedback, and if there turns out to be a server issue we would then raise a Jira for it. Cheers, | ||
| Comment by Santosh Kumar L [ 21/Dec/12 ] | ||
|
We have terminated the cluster so I cannot provide you the output of db.chunks.find( {jumbo:true}) command. | ||
| Comment by Stennie Steneker (Inactive) [ 21/Dec/12 ] | ||
|
Hi Santosh, It looks like your chunks are balanced except on shard3. I expect the size of the chunks is related to your choice of shard key. Perhaps you have some "jumbo chunks" which cannot be split or migrated on shard3. Can you include the output of:
Thanks, | ||
| Comment by Santosh Kumar L [ 21/Dec/12 ] | ||
|
PFB sh.status() output. mongos> sh.status() shards: { "_id" : "shard1", "host" : "<ip:port>" } { "_id" : "shard2", "host" : "<ip:port>" } { "_id" : "shard3", "host" : "<ip:port>" } { "_id" : "shard4", "host" : "<ip:port>" } { "_id" : "shard5", "host" : "<ip:port>" } { "_id" : "shard6", "host" : "<ip:port>" }databases: { "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "test", "partitioned" : true, "primary" : "shard3" } test.usertable chunks: | ||
| Comment by Santosh Kumar L [ 21/Dec/12 ] | ||
|
We are using YCSB jar for inserting data into MongoDB cluster. 1.mongos router ip:port regards, | ||
| Comment by Eliot Horowitz (Inactive) [ 21/Dec/12 ] | ||
|
Can you send the exact way you are running YCSB? | ||
| Comment by Stennie Steneker (Inactive) [ 21/Dec/12 ] | ||
|
Hi Santosh, Can you post the info from sh.status()? Thanks, | ||
| Comment by Santosh Kumar L [ 21/Dec/12 ] | ||
|
I'm inserting data using YCSB framework. | ||
| Comment by Eliot Horowitz (Inactive) [ 21/Dec/12 ] | ||
|
How are you inserting data? |