Background

Design: WRITING-39975. Scope: WRITING-39974. Epic: SPM-4886. Related: WT-18390, AF-19923, AF-19942.

When a follower becomes the leader, WiredTiger copies its pending ingest changes into the stable table. This is the drain. A blind remove written on the follower may be for a key that is not in stable. An earlier version of blind deletes stopped the process on this case, which caused crash loops after restart.

Change

• The drain applies such a remove anyway, which has no effect, and moves on.
• The drain never stops the node because of this, in any build, including diagnostic builds.
• At the end of each table's drain, log one warning if any such removes were found. The warning names the table and gives the number found. A remove of a key that is not there should not happen, so a non-zero number may point to a caller bug.

Statistics

Added in the same PR:

• disagg_step_up_ingest_drain_orphan_remove: step up ingest table drain orphan removes encountered.

Tests

Python tests in the same PR cover:

• Step-up after follower blind removes of keys that are not in stable: no panic or abort, in diagnostic and release builds.
• Step-up after follower blind removes of keys that are in stable: the keys are gone and the statistic stays at zero.
• One warning per table with the correct count, and no warning when no orphan removes exist.
• The statistic counts orphan removes.
• The node restarts cleanly after step-up with orphan removes in the ingest table.

Acceptance criteria

• No orphan remove stops the drain in any build.
• Warning and statistic match the design.
• New tests pass, and fail without the change.

Depends on

Follower blind remove ticket.