-
Type:
Bug
-
Resolution: Fixed
-
Priority:
Critical - P2
-
Affects Version/s: None
-
Component/s: Cursors, Layered Tables
-
None
-
Storage Engines - Foundations
-
163.872
-
None
-
None
When a follower modify's result moves into the tombstone namespace, __clayered_modify_try_ingest() performs a raw btree modify on the ingest table and covers it with an escaped full update in the same transaction. The raw WT_UPDATE_MODIFY remains on the update chain and reconstructs to an unescaped tombstone-namespace value.
The step-up drain (__layered_copy_ingest_table()) walks the ingest table with a version cursor which returns every committed version, including same-transaction intermediates, reconstructing modify values. On the intermediate version, __wt_clayered_ingest_to_stable_value() strips the trailing byte and aborts on WT_ASSERT_ALWAYS: "ingest to stable drain removed a byte other than the escape byte from a value". Without the assert, the drain writes the truncated value to the stable table as a historical version.
Production hit: AF-20552 (mongod 9.0.0-rc1020, WT fe935cb on mongodb-dsc-release-5). Any BSON document whose length has low 16 bits 0x1414 (e.g. 5140 bytes) begins with the tombstone bytes, so a delta update producing such a document on a follower crashes that node's next step-up.
Reproducer: on a follower, insert a value outside the namespace, modify its first two bytes to \x14\x14, commit, step up. The drain aborts as above.
Fix
Decide the storage form before storing anything, so no intermediate version is ever created:
- Add __wt_modify_result_in_ingest_tombstone_namespace(): given the base value and the modify vector, return whether the result begins with \x14\x14 (with size >= 2) and the result size, without materializing the result (reverse position tracing for bytes 0 - 1, forward length arithmetic). Must exactly mirror __wt_modify_apply semantics; add a Catch2 test cross-checking it against real application on randomized vectors.
- In __clayered_modify_try_ingest():
- Base in decode namespace (escaped): decode, apply the vector to a temp buffer, encode, store one full update (existing path, unchanged).
- Base out of namespace, predicted result out of namespace: raw btree modify (existing fast path, unchanged).
- Base out of namespace, predicted result in namespace: apply the vector to a temp buffer, encode, store one full update. No raw modify.
- Buffer sizing: use __wt_modify_max_memsize for the application buffer (intermediate states can exceed both base and result size); result size + 1 for the encode buffer.
Resulting invariant, to be documented at the drain: a stored WT_UPDATE_MODIFY in an ingest table never reconstructs into the tombstone namespace; standard updates are always in encoded form. The drain and its assert need no changes. This also removes the committed-path exposure that WT-18216 assumed impossible and the failure path stores nothing intermediate either.
Tests: the step-up drain reproducer above; test_layered_tombstone_collision.py modify scenarios extended with a step-up drain; Catch2 cross-check for the prediction helper.
- depends on
-
WT-17933 Stop persisting encoded ingest tombstones prefixed values on disk
-
- Closed
-
-
WT-18057 Preserve tombstone encoding across layered cursor modifies
-
- Closed
-
-
WT-18244 Route layered cursor value transfers through encode/decode wrappers - WT-18194 followup
-
- Open
-
-
WT-18154 Add "non-tombstone trailing byte" assert back in
-
- Open
-
-
WT-18216 Make layered modify encoding atomic on failure
-
- Closed
-
- is related to
-
WT-18503 Step-up's legacy stable rebuild writes block_manager=disagg diverging from create-time metadata
-
- Needs Scheduling
-
-
WT-18514 AF-20552 quick mitigation: remove the failing assert
-
- In Code Review
-
-
WT-18216 Make layered modify encoding atomic on failure
-
- Closed
-
- related to
-
WT-18057 Preserve tombstone encoding across layered cursor modifies
-
- Closed
-
-
WT-18503 Step-up's legacy stable rebuild writes block_manager=disagg diverging from create-time metadata
-
- Needs Scheduling
-
-
WT-18516 test/format (disagg mode=switch) ingest drain aborts on 'cbt->ins == NULL': stable btree contains inserts prior to draining
-
- Needs Scheduling
-