Add diagnostics when prepared update resolution fails to find a key

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Fixed
    • Priority: Major - P3
    • WT12.0.0, 9.1.0-rc0
    • Affects Version/s: None
    • Component/s: Transactions
    • None
    • Storage Engines - Transactions
    • 201.271
    • SE Transactions - 2026-08-14
    • 3
    • v9.0, v8.3, v8.0

      Problem

      __wt_btcur_search_prepared() aborts the library when the search for a prepared operation's key returns non-zero:

      ret = __wt_btcur_search(cbt);
      ...
      WT_ASSERT_ALWAYS(
        CUR2S(cursor), ret == 0, "A valid key must exist when resolving prepared updates.");
      

      Every occurrence so far (AF-3556 on 7.0.20, WT-14743, AF-19193 on 8.0.29) has arrived with nothing but this one line plus a stack trace, which is not enough to distinguish the possible causes. Investigation has repeatedly stalled on needing a core dump.

      Two problems with the current diagnostics:

      1. The message asserts a conclusion the check does not support. The assertion fires on any non-zero return, but __wt_btcur_search() can also return WT_ROLLBACK (the app-assist eviction check in __cursor_enter() converts a cache-wait timeout into WT_ROLLBACK) or an I/O / block-manager error from reading a page back in. A cache-overflow rollback and a genuinely missing key are very different bugs, and both are reported as "a valid key must exist".
      2. No state is dumped. Nothing identifies which transaction, which operation, or which key failed, or what the update chain and page look like.

      Proposed work

      1. Report the actual return value, and only claim a missing key when ret == WT_NOTFOUND.
      2. On failure, dump the context needed to classify the occurrence without a core: the transaction id and prepare/commit timestamps and prepared id, whether this is the commit or rollback path, the operation index within the modify list and txn->mod_count, the operation type, its WT_TXN_OP_KEY_REPEATED flag, the dhandle name and btree type, and the key (recno for column store; a bounded hex dump for row store).
      3. Dump the update chain and page state at the search position where one exists, and the current oldest/stable timestamps and oldest transaction id, so the "aborted underneath us" and "advanced oldest" theories can be tested directly from a log.
      4. Consider whether a non-WT_NOTFOUND error deserves separate treatment. Prepared resolution cannot be retried or partially applied, so returning the error to the caller is not obviously safe — but a cache-wait timeout arguably should not reach this path at all. Note the existing asymmetry: __wt_txn_is_blocking() explicitly refuses to roll back prepared transactions, while the cache_max_wait_us path in __wt_evict_app_assist_worker_check() has no such exemption.

      Notes

      Diagnostics only; this does not attempt to fix the underlying failure. See AF-19193 for the current analysis of the most recent occurrence.

            Assignee:
            Chenhao Qu
            Reporter:
            Chenhao Qu
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: