[SERVER-4355] Support multiple parallel chunk migrations for the same collection Created: 22/Nov/11  Updated: 06/Apr/18  Resolved: 20/Jul/16

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Major - P3
Reporter: Greg Studer Assignee: Dianna Hohensee (Inactive)
Resolution: Duplicate Votes: 28
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Duplicate
duplicates SERVER-24853 Refactor Balancer code to use Migrati... Closed
is duplicated by SERVER-11905 why chunk by chunk movement in balanc... Closed
is duplicated by SERVER-22616 Make balancer architecture more scala... Closed
is duplicated by SERVER-22669 CSRS balancer supports parallel migra... Closed
Related
related to SERVER-4537 better protect all sharding admin ope... Closed
Sprint: Sharding 17 (07/15/16), Sharding 18 (08/05/16)
Participants:
Case:

 Description   

With larger clusters, one migration per cluster can be really really slow, a setting to allow the balancer to start one migration per machine may be more helpful in these cases.



 Comments   
Comment by Lucas [ 06/Apr/18 ]

Sorry.

What I mean is the limit of parallel chunks migrations at shard level (multiple chunks migrations in the same shard).

I expressed myself incorrectly. I mean is that would be nice to have not only one migration per shard but several migrations per shard.

Comment by VictorGP [ 06/Apr/18 ]

There are no chunk migrations at shard level... within a shard you have replication, not chunk migrations.

Comment by Lucas [ 06/Apr/18 ]

I don't think that issue should have been closed. For example I still have MANY migration issues when I add a new shard in my database (or remove a existing one). The migration is simply very slow because it was parallel only at cluster level, not shard level.

I have SSD disks, 10Gbps network, 40 cores and MongoDB simply don't use my hardware resources to migrate data faster. Please let's try to improve chunks migrations.

Comment by Dianna Hohensee (Inactive) [ 20/Jul/16 ]

Closing this ticket as the task has been completed by SERVER-24853

Comment by Benoit Labergri [ 20/Apr/16 ]

Agree with all comments here.
Parallelyze the balancing is really necessary. We encounters also the same problem with weeks of balancing before we can increase our collection size even if our machines have the capacity to handle more update traffic. It impact a lot our capacity planning. The more machine we add at a time and the more time we need to balance things.
A tunning of balancing capacity allowed would be good :

  • number of chunk migration allowed in parallel per shard
  • or throughput limit
    or why not do one migration per collection in parallel ?

Our workaround is to create a new collection that we balanced manualy after having added shards and then refeed all data inside so we reduce the time to 2 days or 10 days in the worst case.
Any idea of parallelyzation would be good, not only one per shard but several migration per shard would be better.
Thanks & regards

Comment by Kevin Rice [ 09/Dec/15 ]

This is STILL a problem. it's 2 years later. Might want to work on this.

Note that Cassandra has worked this out. When you add a node to their cluster, it starts copying data immediately and the node is added in, like, maybe an hour or so.

MongoDB cannot expand and shrink easily because this chunk migration thing is a giant huge massive hulking pain in the butt time-wise, it takes too long to move chunks. In a recent deployment, it's slated to take WEEKS to migrate all the chunks. This is unacceptable. The only solution is to dump and reload data? Excuse my frankness, but that's NOT an 'Enterprise Grade' solution.

Comment by Kevin J. Rice [ 28/Feb/13 ]

Limiting to one migration per machine can be troublesome since we may have different mount points per shard if there is more than one shard per machine.

IIRC, there's no need to be too smart about this with excessive locking. In a 50 shard Mongo cluster, you'd have 49 pairs of adjacent nodes, and a persistent daemon / subprocess / thread could be spawned to handle the balancing of those two nodes. Every once in a while (minute? 5 minutes?), the daemon could wake up, compare chunk counts, and either go to sleep or go to work.

Visibility of the balancer state is of significant concern. Therefore, I'd suggest the balancer write to a capped collection with what each node-pair balancer is doing, indexed by 'fromShard:toShard' or something.

Part of the problem here is having balancers on mongos routers; it seems like this code belongs somewhere else, in an independent process, with parameters for how aggressive it should be as well as the existing ones of which hours it should run, etc.

Aggressiveness is important because if we're writing to a single shard so much that there is no spare IO available for a long time, it can never migrate work to other shards to balance the load better. Some kind of radioactive-decay curve should apply so that the longer it goes unbalanced, the more important balancing is, until finally balancing is more important than writing new data and you can get a chunk actually moved despite the overload.

Just some ideas.

Generated at Thu Feb 08 03:05:45 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.