-
Type:
Task
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Statistics
-
Storage Engines - Foundations
-
31.785
-
None
-
5
Include a step-down dashboard for disaggregated storage in Grafana.
Two statistics do most of the work:
- txn_stepdown_ts_set – reads 1 from the moment the step-down timestamp is set until the node is a follower, and 0 otherwise, so a 0 -> 1 transition marks one step-down attempt.
- disagg_role_leader – on a reconfigure this only reaches 0 from inside the step-down itself, so a 1 -> 0 transition marks one completed step-down.
disagg_step_down_time is written only on the success path, so a change in its value is a third, independent success marker.
Stat keys below are the WiredTiger names for grepping dist/stat_data.py; FTDC / serverStatus keys are the description text under the prefix section.
Risks
- How long the step-down call took – disagg_step_down_time ("step down most recent time (msecs)"). (maybe we can also time from when stepdown_ts was set and when we became follower); or we derive this from server somehow?
- Failed step-downs – txn_stepdown_ts_set 0 -> 1 transitions with no matching disagg_role_leader 1 -> 0. A failed demotion panics, so these show up as an attempt whose series ends rather than completing.
- Nodes that crashed within 30 seconds of a step-down – a txn_stepdown_ts_set 0 -> 1 or disagg_role_leader 1 -> 0 transition followed within 30s by a serverStatus.uptime reset.
- Cache pressure, step-down timestamp set vs unset – cache_bytes_inuse / cache_bytes_max, plotted separately for txn_stepdown_ts_set at 1 and at 0, as both an average and a P99.
- Stable not advancing to the step-down timestamp – txn_global_stable_timestamp restricted to when txn_stepdown_ts_set is 1, plus how long txn_stepdown_ts_set stays at 1 as the wait duration.
- Rolled-back transactions – txn_rollback_stepdown. It only ever increases, so a 24h increase() grouped by cluster gives the "which clusters had the most rollbacks" view directly.
Rewards
- Nodes successfully stepping down, with a total for the success percentage – successes from disagg_role_leader 1 -> 0 transitions, total from txn_stepdown_ts_set 0 -> 1 transitions, percentage as the ratio.
- Read performance remaining high – layered_curs_search, layered_curs_search_near, layered_curs_next, layered_curs_prev. Should stay flat across the whole period.
- Write performance high until reconfigure – layered_curs_insert, layered_curs_update, layered_curs_remove, layered_curs_modify. Expect flat until the reconfigure, then zero; a dip before the reconfigure is the signal to alert on.
- Schema operations while the step-down timestamp is set – session_table_create_success and session_table_create_fail, session_table_drop_success and session_table_drop_fail, session_table_publish_success and session_table_publish_fail, restricted to when txn_stepdown_ts_set is 1.