[SERVER-2218] Make sharding balancer smarter when asymmetric hardware Created: 14/Dec/10  Updated: 17/Nov/23

Status: Backlog
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Keith Branton Assignee: Backlog - Cluster Scalability
Resolution: Unresolved Votes: 13
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

All


Issue Links:
Related
Assigned Teams:
Cluster Scalability
Participants:

 Description   

Make the balancer shard size aware, and design it to use all available storage.

e.g. If a shard set currently consisting of shards with 2TB capacity is extended by adding some 3TB shards, only 2TB of the larger shards will currently be used.

Given that sharding is mongo's "scale out" strategy, this would be a major improvement in allowing projects to scale out much more economically.



 Comments   
Comment by Wernfried Domscheit [ 16/Apr/21 ]

Zone Sharding could be an option to achieve this requirement. However, you need to know the distribution of values in your sharding-key.

 

 

Comment by MediaMath Mongo [ 16/Apr/12 ]

We would love to have user configurable "ratio" of chunks per shard or per tag in case you have to many shards to deal with. This way with chunk size and maxSize, we should be able to control how much load each shard should handle without disabling balancer or care about disk utilization (why should we anyway). This should also help with new shard migration without performance degradation.

Comment by Dwight Merriman [ 09/Feb/12 ]

maxSize is the partial solution for the moment

Comment by Keith Branton [ 17/Jan/11 ]

So I raised this ticket as a result of the discussion here: http://groups.google.com/group/mongodb-user/browse_thread/thread/ad8a77f6cd7ff6d3/a5717d5d037a8cf1?lnk=gst&q=keith+branton+3tb#a5717d5d037a8cf1

but maybe misunderstood Eliot's response, since reading http://www.mongodb.org/display/DOCS/Configuring+Sharding section titled maxSize it seems that the balancer already does what I would want - i.e. use all the disk space available on each shard.

Eliot said: "Balancing isn't terribly smart when it comes to disk right now and
generally assumes each shard has the same capabilities.
We'll be enhancing this over time, but for now if you can keep shards
consistent, that would be better. "

and

"Not sure there is a case just for that, feel free to add an explicit one. "

So this is really the case to track whatever enhancement Eliot had in mind

Generated at Thu Feb 08 02:59:19 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.