Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16802

Order of balancer chunk moves depends on order of config.collections

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: 2.8.0-rc4
    • Fix Version/s: 3.4.15, 3.6.4, 3.7.4
    • Component/s: Sharding
    • Labels:
    • Backwards Compatibility:
      Fully Compatible
    • Backport Requested:
      v3.6, v3.4
    • Sprint:
      Sharding 2018-02-26, Sharding 2018-03-12, Sharding 2018-03-26
    • Case:

      Description

      Background

      Currently, Balancer::_doBalanceRound() and BalancerPolicy::balance() together find candidate chunks with the following pseudocode:

      • For each collection in config.collections:
        • Look for chunks on draining shards
        • Look for tag violating chunks
        • Look for imbalance within each tag
      Problem

      This approach means that the balancer can attempt "regular" imbalance chunk moves before tag violation chunk moves, which can be before shard drain moves. This is counter-intuitive, because users expect:

      • chunks to commence moving off draining shards very soon after removeShard has been run, and
      • tag violation chunks to be moved ahead of regular imbalance moves.
      Impact

      This is worsened by:

      1. A lot of imbalanced sharded collections, but a shard being drained has chunks only in collections near the end of the list (e.g. caused by tags). In this case, the user will observe many "irrelevant" non-draining chunk moves prior to (and in between) draining chunk moves (the pattern repeats each balancing round).
      2. Where earlier lower-priority moves can't complete because the TO shard can't satisfy the w:majority pre-commit check. Then each problematic earlier chunk move fails after 10 hours, causing higher-priority moves to happen a lot later. In the case of a shard drain, this can make it take a very long time, and the system to outwardly appear nearly idle (very little balance/drain/move activity). Again, this is counter-intuitive because the user believes that they have just instructed the system to start draining (but it "isn't").
      Suggestion

      Perhaps the code could be rearranged along the lines of:

      • For each collection in config.collections:
        • Look for chunks on draining shards
      • For each remaining collection:
        • Look for tag violating chunks
      • For each remaining collection:
        • Look for imbalance within each tag

      Each collection would still have at most 1 chunk move per balance round, which should allow lower-priority moves to make progress (e.g. the balancer won't be "hogged" when draining a large shard, or when adding new tags). But it would also have the benefit of ensuring that — irrespective of the order of config.collections — draining moves are given priority over tag violation moves, which are given priority over imbalance moves.

        Attachments

          Activity

            People

            Assignee:
            kevin.pulo Kevin Pulo
            Reporter:
            kevin.pulo Kevin Pulo
            Participants:
            Votes:
            1 Vote for this issue
            Watchers:
            10 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: