Investigate: multi-statement transaction may read silently incomplete results after shard catalog history filtering

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Catalog and Routing
    • ALL
    • 🟦 Shard Catalog
    • None
    • None
    • None
    • None
    • None
    • None

      Suspected problem

      When featureFlagShardCatalogExpiredHistoryFiltering is enabled, authoritative metadata disk recovery drops chunk documents unless shard == <me> or onCurrentShardSince > <WT oldest timestamp> ( readCollectionMetadataFromDisk() in +[collection_metadata_synchronizer | https://github.com/10gen/mongo/blob/master/src/mongo/db/shard_role/shard_catalog/collection_metadata_synchronizer.cpp#L37-L144), and rebuilds the routing table with RoutingTableHistory::makeNewAllowingGaps.

      Multi-statement transactions keep their storage snapshot open across statements, while each statement acquires a fresh ownership filter from the currently installed metadata. The oldest timestamp does not account for open transactions, so a transaction's read timestamp can fall below it mid-transaction.

      If that holds, documents the shard owned at the read timestamp would be discarded as orphans -  partial results, converting today's loud failure StaleChunkHistory into a silent one.

      Suspected scenario

      With minSnapshotHistoryWindowInSeconds < transactionLifetimeLimitSeconds:

      • A snapshot transaction starts on shard S with read time T.
      • Chunk C migrates S -> R at T1 > T. The router still targets S for C's range at T.
      • Oldest timestamp advances past T1 while the transaction stays open.
      • Filtering metadata is cleared on the primary (e.g. failed migration) and disk recovery installs a table without C.
      • The transaction's next statement on C's range at atClusterTime = T passes the placement version check (S's own chunks are retained) and silently loses C's documents.

      Investigation

      • Confirm the scenario end-to-end, ideally with a failpoint-driven repro.
      • Check whether existing transaction-vs-migration protections fire anywhere in
          this path.

      Possible fix directions

      • Bound the filtering predicate by transactionLifetimeLimitSeconds /  WT pinned timestamp so chunks needed by open snapshots are retained. Note: subtracting transactionLifetimeLimitSeconds from the bound might not be sifficient.

      Acceptance criteria

      • The suspected scenario is confirmed or ruled out.
      • If confirmed: reads below the filtering bound either see complete results or fail with a retryable error - never partial results.

       

            Assignee:
            Unassigned
            Reporter:
            Tymoteusz Slepowronski (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: