Background

Currently, Balancer::_doBalanceRound() and BalancerPolicy::balance() together find candidate chunks with the following pseudocode:

• For each collection in config.collections:
  • Look for chunks on draining shards
  • Look for tag violating chunks
  • Look for imbalance within each tag

Problem

This approach means that the balancer can attempt "regular" imbalance chunk moves before tag violation chunk moves, which can be before shard drain moves. This is counter-intuitive, because users expect:

• chunks to commence moving off draining shards very soon after removeShard has been run, and
• tag violation chunks to be moved ahead of regular imbalance moves.

Impact

This is worsened by:

1. A lot of imbalanced sharded collections, but a shard being drained has chunks only in collections near the end of the list (e.g. caused by tags). In this case, the user will observe many "irrelevant" non-draining chunk moves prior to (and in between) draining chunk moves (the pattern repeats each balancing round).
2. Where earlier lower-priority moves can't complete because the TO shard can't satisfy the w:majority pre-commit check. Then each problematic earlier chunk move fails after 10 hours, causing higher-priority moves to happen a lot later. In the case of a shard drain, this can make it take a very long time, and the system to outwardly appear nearly idle (very little balance/drain/move activity). Again, this is counter-intuitive because the user believes that they have just instructed the system to start draining (but it "isn't").

Suggestion

Perhaps the code could be rearranged along the lines of:

• For each collection in config.collections:
  • Look for chunks on draining shards

• For each remaining collection:
  • Look for tag violating chunks

• For each remaining collection:
  • Look for imbalance within each tag

Each collection would still have at most 1 chunk move per balance round, which should allow lower-priority moves to make progress (e.g. the balancer won't be "hogged" when draining a large shard, or when adding new tags). But it would also have the benefit of ensuring that — irrespective of the order of config.collections — draining moves are given priority over tag violation moves, which are given priority over imbalance moves.