Routed reads on a hashed-sharded collection fail with persistent StaleConfig (13388) "this shard no longer contains chunks" after restore on master; flushRouterConfig does not clear it

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Incomplete
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Catalog and Routing
    • ALL
    • v9.0
    • CAR Team 2026-08-03
    • 0
    • 🟦 Shard Catalog
    • None
    • None
    • None
    • None
    • None
    • None

      Note: AI investigation + notes. Please re-assign as needed.

       

      On mongodb-mongo-master (nightly), after a cross-project direct-attach point-in-time restore of a 2-shard Atlas cluster, every routed read of a hashed-sharded collection (food.dairy, shard key {_id: "hashed"}) fails with error 13388:

        Exceeded maximum number of 10 retries attempting '<op>' :: got stale shardVersion
        response from shard <shard-0> :: this shard no longer contains chunks for food.dairy,
        the collection may have been dropped

      Reproduced with validate (full:true), collStats, and ClusterAggregate::runAggregate (countDocuments). The config server and shard-0 disagree on the placement version: router/config sends v:{1,0} (shard-0 owns chunks), shard-0 responds v:{0,0} (owns none). Epoch and timestamp match; only the placement version differs.

      Key detail: flushRouterConfig returns {flushed: true, ok: 1} but the immediately following read still fails with the same error, so this is not a stale mongos router cache. The config-vs-shard disagreement is authoritative and does not converge across the driver's 10 internal retries plus external retries over ~2+ minutes. Unsharded collections in the same cluster read fine.

      Version isolation: the identical workload on released 8.0 passes; only CONTINUOUS_DELIVERY (nightly master) fails. Regression window ~2026-07-09 to 2026-07-12. Possibly related to SERVER-91465 and the BF-44022 / SERVER-130902 StaleConfig family (same 13388 error on master).

      Ask: please confirm whether this is a known/expected nightly behavior change or a regression in validate/StaleConfig routing on sharded collections following a restore.

      Filed by the Atlas Backups (CPS) team; surfaced by an Atlas gov-cloud disk-snapshot restore e2e (CLOUDP-425975).

      Evergreen refs:

            Assignee:
            Pol Pinol
            Reporter:
            Jason Davis
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: