[SERVER-2218] Make sharding balancer smarter when asymmetric hardware Created: 14/Dec/10 Updated: 17/Nov/23 |
|
| Status: | Backlog |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Keith Branton | Assignee: | Backlog - Cluster Scalability |
| Resolution: | Unresolved | Votes: | 13 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
All |
||
| Issue Links: |
|
||||
| Assigned Teams: |
Cluster Scalability
|
||||
| Participants: | |||||
| Description |
|
Make the balancer shard size aware, and design it to use all available storage. e.g. If a shard set currently consisting of shards with 2TB capacity is extended by adding some 3TB shards, only 2TB of the larger shards will currently be used. Given that sharding is mongo's "scale out" strategy, this would be a major improvement in allowing projects to scale out much more economically. |
| Comments |
| Comment by Wernfried Domscheit [ 16/Apr/21 ] |
|
Zone Sharding could be an option to achieve this requirement. However, you need to know the distribution of values in your sharding-key.
|
| Comment by MediaMath Mongo [ 16/Apr/12 ] |
|
We would love to have user configurable "ratio" of chunks per shard or per tag in case you have to many shards to deal with. This way with chunk size and maxSize, we should be able to control how much load each shard should handle without disabling balancer or care about disk utilization (why should we anyway). This should also help with new shard migration without performance degradation. |
| Comment by Dwight Merriman [ 09/Feb/12 ] |
|
maxSize is the partial solution for the moment |
| Comment by Keith Branton [ 17/Jan/11 ] |
|
So I raised this ticket as a result of the discussion here: http://groups.google.com/group/mongodb-user/browse_thread/thread/ad8a77f6cd7ff6d3/a5717d5d037a8cf1?lnk=gst&q=keith+branton+3tb#a5717d5d037a8cf1 but maybe misunderstood Eliot's response, since reading http://www.mongodb.org/display/DOCS/Configuring+Sharding section titled maxSize it seems that the balancer already does what I would want - i.e. use all the disk space available on each shard. Eliot said: "Balancing isn't terribly smart when it comes to disk right now and and "Not sure there is a case just for that, feel free to add an explicit one. " So this is really the case to track whatever enhancement Eliot had in mind |