-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Cursors
-
Storage Engines - Foundations
-
294.542
-
SE Foundations - 2026-08-04, SE Foundations - 2026-08-18
-
5
-
1
Summary
__clayered_assert_stable_mode() in src/cursor/cur_layered.c asserts that the stable cursor’s btree is read-write for a leader and read-only for a follower. This invariant holds in steady state but is racy during step-down, so the assertion can fire spuriously on diagnostic builds.
Assertion
{{/* The stable cursor's btree must be read-write for a leader and read-only for a follower. */WT_ASSERT(CUR2S(clayered),
(clayered->last_role == WTI_CLAYERED_ROLE_LEADER) !=
F_ISSET(CUR2BT(clayered->stable_cursor), WT_BTREE_READONLY));}}
Race description
_disagg_mark_btrees_readonly_then_step_down() (in src/conn/conn_layered.c) first marks live disaggregated btrees WT_BTREE_READONLY, and only afterwards flips conn->layered_table_manager.leader = false. A layered cursor samples the leader flag locklessly in _clayered_enter(); if it samples between those two steps it still sees role == LEADER while its live stable btree is already read-only, so the assertion fails. The follower direction is safe: followers always open the stable table at a checkpoint URI, and checkpoint dhandles are unconditionally read-only.
How it was found
Exposed by the async step-down work in test/format (WT-18044):
- Worker threads continue read operations while a background thread runs reconfigure("disaggregated=(role=follower)").
- Layered cursors can therefore enter during the step-down window.
- The previous synchronous path joined all workers before reconfiguring and never exercised this window.
Impact
- No correctness issue: no data is read or written incorrectly.
- The window only allows a leader-role cursor to observe its (already write-quiesced) live stable btree as read-only.
- The actual read path behaves correctly, and once the role change is visible the cursor will reopen the stable table on the next operation.
- Symptom: a spurious WT_ASSERT abort on diagnostic builds during step-down.
- duplicates
-
WT-16879 Fix the potential data race between open btree or open dhandle and primary step down
-
- Closed
-
- is related to
-
WT-18044 Add write-pause concurrency in test/format and make disagg step-down fully asynchronous
-
- Blocked
-
- related to
-
WT-18142 failed: format-stress-test-disagg-switch-1 Checkpoint Cleanup issues a writes as a follower
-
- Blocked
-