Context

WT-17989 identified that step-up currently drains an ingest table and then clears it with a whole-table non-transactional truncate. The clear walks every ingest key while holding one read snapshot. For a large ingest table under cache pressure, the scan can pin the global oldest transaction ID long enough to prevent update reclamation, leave eviction stuck, and receive WT_ROLLBACK with WT_OLDEST_FOR_EVICTION.

WT-17989 uses a temporary standalone mitigation that sets WT_SESSION_IGNORE_CACHE_SIZE during the final clear. This ticket tracks the long-term design that removes the long-lived snapshot and releases completed ingest pages incrementally.

Goal

Replace the drain-then-whole-table-clear workflow with a single incremental row-store drain that completes one key and one source leaf at a time. The drain must release its read snapshot after each key and hand a completed non-root ingest leaf to eviction as soon as the walk crosses its boundary.

Proposed MVP

• Scope the first implementation to BTREE_ROW.
• Preserve the existing fail-fast behavior. Do not add drain retry, resume, or a persistent recovery frontier.
• Snapshot the truncate list into worker-owned entries sorted by timestamp and transaction ID before workers start. Keep per-drain frontier state separate from the shared truncate-list objects.
• Walk the ingest tree once in key order.
• For each ingest key:
  • Replay stable-only portions of covering truncate ranges in timestamp order, partitioning each closed truncate range around visited ingest keys.
  • Read and merge the stable key's version chain with ingest versions and covering truncate tombstones, preserving the existing total version order and prepared-update behavior.
  • Finish all stable-side work, reset borrowed cursor state, localize values, and explicitly release the read snapshot. Verify the drain does not carry one pinned_id across keys.
  • Apply the globally visible non-transactional tombstone to the ingest source key.
  • Advance truncate frontier state only after the stable chain and source tombstone both complete. Pre-grow any required key buffers so no fallible allocation occurs after source deletion.
• Hold an extra hazard on the current non-root ingest leaf. When the walk crosses a leaf boundary, reset the completed leaf's prune gate, mark it EVICT_SOON, consume the drain hazard in a direct no-split eviction attempt, and accept EBUSY while preserving sticky eviction intent.
• At normal EOF, finish the final leaf, replay only remaining truncate-range remainders, release the EOF snapshot, and clear the shared truncate list after the whole work item succeeds.
• Remove the trailing whole-table ingest clear and diagnostic empty-table scan from the successful drain path.

Correctness invariants

• Stable-side work for a key completes before the irreversible ingest source tombstone.
• A truncate's frontier names only the last ingest key whose stable chain and source tombstone both completed.
• Visited ingest keys are handled exactly once by inline version merging; stable-only keys are handled exactly once by restricted truncate replay.
• Snapshot release occurs only after all cursors have released borrowed values.
• A completed leaf is never accessed after its drain hazard is transferred to eviction.
• Errors retain the current connection-fatal semantics; recovery starts from the existing durable checkpoint boundary rather than resuming a partial drain.

Definition of done

• The original pinned WT-17989 format reproduction passes without WT_SESSION_IGNORE_CACHE_SIZE and without WT_ROLLBACK retry.
• Deterministic tests verify per-key snapshot release and that global oldest can advance between consecutive drained keys.
• Multi-leaf tests verify completed pages are evicted or left marked EVICT_SOON before later leaves finish, including other-hazard and competing-eviction EBUSY cases.
• Tests cover stable/ingest version ordering, prepared updates, empty logical streams, overlapping truncates, inclusive range endpoints, and stable-only truncate gaps.
• Tests verify EOF cleanup, real-error ownership transfer, no trailing source scan, and no page split during direct eviction.
• Cache-residency measurements demonstrate that completed leaves are released incrementally rather than retained until whole-table completion.
• Relevant Python/csuite coverage and x86/arm64 disaggregated format stress variants pass.