Issue Summary

While testing the backport of WT-16698 on the 7.0 branch, an assertion failure was encountered during a checkpoint operation in WiredTiger. The error message is:

[1773725438:760917][85578:0x209a8f100], t, file:T00008.wt, WT_SESSION.checkpoint: [WT_VERB_DEFAULT][ERROR]: void __wt_tree_modify_set(WT_SESSION_IMPL *), 795: WiredTiger assertion failed: '!((((session)->flags) & (0x40000u)) != 0)'. A btree is marked dirty during RTS
[1773725438:760934][85578:0x209a8f100], t, file:T00008.wt, WT_SESSION.checkpoint: [WT_VERB_DEFAULT][ERROR]: void __wt_abort(WT_SESSION_IMPL *), 28: aborting WiredTiger library
zsh: abort      ./t -c CONFIG.stress

Context

• The test was run with a complex configuration involving multiple tables and various WiredTiger parameters (see full config in Slack thread).
• The issue appears related to the btree being marked dirty during RTS (Rollback to Stable).
• The changes under test are from WT-16698, commit b2824f4179dc3d008de6d8f26089f0520706fdfb.
• This error aborts the WiredTiger library, preventing further testing.

Proposed Solution

• Investigate the assertion in __wt_tree_modify_set to determine why the btree is marked dirty during RTS checkpoint.
• Review the changes in WT-16698 and their interaction with checkpoint and RTS logic.
• Reproduce and debug with the provided configuration to identify the root cause.
• Provide a fix or workaround to prevent the assertion failure and allow successful checkpointing after RTS.

Original Slack thread
This ticket was generated by AI from a Slack thread.