Decommissioning a removed shard can cause CMC to hit a tassert about a collection disappearing

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fixed
    • Priority: Major - P3
    • 9.1.0-rc0, 9.1.0-rc1022
    • Affects Version/s: None
    • Component/s: None
    • None
    • Catalog and Routing
    • Fully Compatible
    • ALL
    • CAR Team 2026-09-14
    • 0
    • 🟩 Routing and Topology
    • None
    • None
    • None
    • None
    • None
    • None

      This issue is very similar to SERVER-96906 except that the drop of the collections does not come from the removeShard operation but rather from the shard being cleaned up prior to being decommissioned.

      The add_remove_shards.py hook deletes the user databases on the shard prior to shutting it down. These drops are done via direct shard connection and therefore do not acquire the DDL lock or critical section and so do not serialize with any cluster level DDLs.

      If a CMC command starts before a shard has updated its shard registry and still targets this shard, then it can see the result of the dropDatabases which can result in failures or false positive inconsistencies.

            Assignee:
            Allison Easton
            Reporter:
            Allison Easton
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: