[SERVER-4355] Support multiple parallel chunk migrations for the same collection Created: 22/Nov/11 Updated: 06/Apr/18 Resolved: 20/Jul/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | New Feature | Priority: | Major - P3 |
| Reporter: | Greg Studer | Assignee: | Dianna Hohensee (Inactive) |
| Resolution: | Duplicate | Votes: | 28 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||
| Sprint: | Sharding 17 (07/15/16), Sharding 18 (08/05/16) | ||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||||||||||||||||||
| Description |
|
With larger clusters, one migration per cluster can be really really slow, a setting to allow the balancer to start one migration per machine may be more helpful in these cases. |
| Comments |
| Comment by Lucas [ 06/Apr/18 ] |
|
Sorry. What I mean is the limit of parallel chunks migrations at shard level (multiple chunks migrations in the same shard). I expressed myself incorrectly. I mean is that would be nice to have not only one migration per shard but several migrations per shard. |
| Comment by VictorGP [ 06/Apr/18 ] |
|
There are no chunk migrations at shard level... within a shard you have replication, not chunk migrations. |
| Comment by Lucas [ 06/Apr/18 ] |
|
I don't think that issue should have been closed. For example I still have MANY migration issues when I add a new shard in my database (or remove a existing one). The migration is simply very slow because it was parallel only at cluster level, not shard level. I have SSD disks, 10Gbps network, 40 cores and MongoDB simply don't use my hardware resources to migrate data faster. Please let's try to improve chunks migrations. |
| Comment by Dianna Hohensee (Inactive) [ 20/Jul/16 ] |
|
Closing this ticket as the task has been completed by |
| Comment by Benoit Labergri [ 20/Apr/16 ] |
|
Agree with all comments here.
Our workaround is to create a new collection that we balanced manualy after having added shards and then refeed all data inside so we reduce the time to 2 days or 10 days in the worst case. |
| Comment by Kevin Rice [ 09/Dec/15 ] |
|
This is STILL a problem. it's 2 years later. Might want to work on this. Note that Cassandra has worked this out. When you add a node to their cluster, it starts copying data immediately and the node is added in, like, maybe an hour or so. MongoDB cannot expand and shrink easily because this chunk migration thing is a giant huge massive hulking pain in the butt time-wise, it takes too long to move chunks. In a recent deployment, it's slated to take WEEKS to migrate all the chunks. This is unacceptable. The only solution is to dump and reload data? Excuse my frankness, but that's NOT an 'Enterprise Grade' solution. |
| Comment by Kevin J. Rice [ 28/Feb/13 ] |
|
Limiting to one migration per machine can be troublesome since we may have different mount points per shard if there is more than one shard per machine. IIRC, there's no need to be too smart about this with excessive locking. In a 50 shard Mongo cluster, you'd have 49 pairs of adjacent nodes, and a persistent daemon / subprocess / thread could be spawned to handle the balancing of those two nodes. Every once in a while (minute? 5 minutes?), the daemon could wake up, compare chunk counts, and either go to sleep or go to work. Visibility of the balancer state is of significant concern. Therefore, I'd suggest the balancer write to a capped collection with what each node-pair balancer is doing, indexed by 'fromShard:toShard' or something. Part of the problem here is having balancers on mongos routers; it seems like this code belongs somewhere else, in an independent process, with parameters for how aggressive it should be as well as the existing ones of which hours it should run, etc. Aggressiveness is important because if we're writing to a single shard so much that there is no spare IO available for a long time, it can never migrate work to other shards to balance the load better. Some kind of radioactive-decay curve should apply so that the longer it goes unbalanced, the more important balancing is, until finally balancing is more important than writing new data and you can get a chunk actually moved despite the overload. Just some ideas. |