Layered modify stores an unescaped intermediate version; step-up drain aborts on it

XMLWordPrintableJSON

    • Storage Engines - Foundations
    • 163.872
    • None
    • None

      When a follower modify's result moves into the tombstone namespace, __clayered_modify_try_ingest() performs a raw btree modify on the ingest table and covers it with an escaped full update in the same transaction. The raw WT_UPDATE_MODIFY remains on the update chain and reconstructs to an unescaped tombstone-namespace value.

      The step-up drain (__layered_copy_ingest_table()) walks the ingest table with a version cursor which returns every committed version, including same-transaction intermediates, reconstructing modify values. On the intermediate version, __wt_clayered_ingest_to_stable_value() strips the trailing byte and aborts on WT_ASSERT_ALWAYS: "ingest to stable drain removed a byte other than the escape byte from a value". Without the assert, the drain writes the truncated value to the stable table as a historical version.

      Production hit: AF-20552 (mongod 9.0.0-rc1020, WT fe935cb on mongodb-dsc-release-5). Any BSON document whose length has low 16 bits 0x1414 (e.g. 5140 bytes) begins with the tombstone bytes, so a delta update producing such a document on a follower crashes that node's next step-up.

      Reproducer: on a follower, insert a value outside the namespace, modify its first two bytes to \x14\x14, commit, step up. The drain aborts as above.

      Fix

      Decide the storage form before storing anything, so no intermediate version is ever created:

      • Add __wt_modify_result_in_ingest_tombstone_namespace(): given the base value and the modify vector, return whether the result begins with \x14\x14 (with size >= 2) and the result size, without materializing the result (reverse position tracing for bytes 0 - 1, forward length arithmetic). Must exactly mirror __wt_modify_apply semantics; add a Catch2 test cross-checking it against real application on randomized vectors.
      • In __clayered_modify_try_ingest():
        • Base in decode namespace (escaped): decode, apply the vector to a temp buffer, encode, store one full update (existing path, unchanged).
        • Base out of namespace, predicted result out of namespace: raw btree modify (existing fast path, unchanged).
        • Base out of namespace, predicted result in namespace: apply the vector to a temp buffer, encode, store one full update. No raw modify.
      • Buffer sizing: use __wt_modify_max_memsize for the application buffer (intermediate states can exceed both base and result size); result size + 1 for the encode buffer.

      Resulting invariant, to be documented at the drain: a stored WT_UPDATE_MODIFY in an ingest table never reconstructs into the tombstone namespace; standard updates are always in encoded form. The drain and its assert need no changes. This also removes the committed-path exposure that WT-18216 assumed impossible and the failure path stores nothing intermediate either.

      Tests: the step-up drain reproducer above; test_layered_tombstone_collision.py modify scenarios extended with a step-up drain; Catch2 cross-check for the prediction helper.

            Assignee:
            Yury Ershov
            Reporter:
            Yury Ershov
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: