Investigate long metadata recovery time window with auth shards

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Works as Designed
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Catalog and Routing
    • v9.0
    • CAR Team 2026-07-20, CAR Team 2026-08-03
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      SERVER-130572 introduced an FSM workload running multiple DDLs and CRUD operations concurrently while enabling/disabling the replica set writes. 
      Workers issue reads with maxTimeMS set to 2s. The test fails with MaxTimeMSExpired error.

      The metadata refresh blocked long enough that reads on hit maxTimeMS even though write blocking does not reject reads. 
      Metadata recovery spends most of its time in _waitForConfigTimeOrChunkVersionChange, not in disk I/O. 
      For example, in the first test failure reported below, the disk read completes in about 47 ms, while the full recovery completes in 1508 ms. This makes the read to hit maxTimeMSExpired as reported in the logs:

      [j0:s1:prim] | 2026-07-16T18:40:14.896+00:00 I SHARDING 22062 [S] [conn1150] "Failed to refresh metadata for collection","attr": { namespace: "test2_fsmdb0.ddl_coll_6", error: "MaxTimeMSExpired: operation exceeded time limit" }

       
      Test failure examples: this and this (reporting 
      makeNoopWriteToAdvanceClusterTimeImpl logs)). The related patches report more examples of the failure.

            Assignee:
            Aitor Esteve Alvarado
            Reporter:
            Anna Maria Nestorov
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: