Metadata recovery tied to interruptible operation can prevent filtering metadata from ever being installed

XMLWordPrintableJSON

    • Catalog and Routing
    • Fully Compatible
    • ALL
    • v9.0
    • CAR Team 2026-07-20, CAR Team 2026-08-03
    • 200
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      When an operation encounters unknown sharding metadata, it must step back and trigger a recovery process to install the ownership metadata from the shard before retrying. Only one thread should be responsible for performing this recovery; all other threads that need the same metadata are expected to wait for that recovery to complete and then reuse its result.

      Current behavior (9.0 FCV, authoritative shards):

      The first operation that encounters the unknown metadata becomes the one responsible for performing the recovery itself, inline. This has a significant drawback: if that operation is interrupted (e.g. due to maxTimeMS), the recovery fails outright. Since the recovery was tied to that specific operation rather than to an independent, non-killable task, the work is lost and cannot be reused by the other waiting threads. Each waiting thread must then start its own recovery from scratch, only to risk the same fate.

      In practice, this can lead to a situation where no operation is ever able to successfully complete the metadata recovery, effectively causing a livelock.

      Previous behavior (8.0 FCV):

      The first operation to hit unknown metadata spawns a separate, non-killable thread dedicated to performing the recovery. All other operations simply wait on this thread to finish. Because the recovery thread is initiated by the server itself (not tied to any particular client operation), maxTimeMS only affects the operations waiting on the result, it has no effect on the recovery thread, which runs to completion regardless of interruptions to the callers.

            Assignee:
            Pol Pinol
            Reporter:
            Pol Pinol
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: