Avoid slow removeShard in *_with_balancer_and_config_transitions_and_add_remove_shard

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: 8.0.0, 8.3.0, 9.1.0-rc0, 9.0.0
    • Component/s: None
    • None
    • Catalog and Routing
    • CAR Team 2026-09-14
    • 0
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      [Test only]

       

      Problem

      The *_with_balancer_and_config_transitions_and_add_remove_shard test suites have dynamics that lead to slow shard removal and eventually timeout in slow variants:

      1. Random migrations continuously create more chunks for config.system.sessions.
      2. config.system.sessions isn't cleaned between tests --> Ever growing chunk count.
      3. removeShard needs to drain that ever growing pile of chunks out.
      4. Previous removeShard drainings also tend to leave the chunks unevenly balanced.
      5. The balancer in "random migrations" test-only mode prefers moveCollection (resharding) over draining 50% of the time, by default.

       

      Potential solutions and mitigations

      1. Avoid random migration for config.system.sessions, or limit the number of chunk splits.
      2. Drop config.system.sessions in background or between tests.
      3. Merge config.system.sessions chunks before removeShard (note: may be problematic since chunk merges discarding the chunk history).
      4. Adjust skipMoveCollectionThreshold so the balancer favors draining.
      5. Change the balancer random migrations mode so it always prefers draining over moveCollection.

       

      (2) seems the best long-term solution, but needs SERVER-133099 to avoid generating much tech debt.

      (4) seems the simplest short term mitigation but skipMoveCollectionThreshold is only respected since SERVER-133663.

            Assignee:
            Joan Bruguera Micó
            Reporter:
            Joan Bruguera Micó
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: