Add step-down observability metrics to Grafana

XMLWordPrintableJSON

    • Storage Engines - Foundations
    • 31.785
    • None
    • 5

      Include a step-down dashboard for disaggregated storage in Grafana.

      Two statistics do most of the work:

      • txn_stepdown_ts_set – reads 1 from the moment the step-down timestamp is set until the node is a follower, and 0 otherwise, so a 0 -> 1 transition marks one step-down attempt.
      • disagg_role_leader – on a reconfigure this only reaches 0 from inside the step-down itself, so a 1 -> 0 transition marks one completed step-down.

      disagg_step_down_time is written only on the success path, so a change in its value is a third, independent success marker.

      Stat keys below are the WiredTiger names for grepping dist/stat_data.py; FTDC / serverStatus keys are the description text under the prefix section.

      Risks

      1. How long the step-down call tookdisagg_step_down_time ("step down most recent time (msecs)"). (maybe we can also time from when stepdown_ts was set and when we became follower); or we derive this from server somehow?
      2. Failed step-downstxn_stepdown_ts_set 0 -> 1 transitions with no matching disagg_role_leader 1 -> 0. A failed demotion panics, so these show up as an attempt whose series ends rather than completing.
      3. Nodes that crashed within 30 seconds of a step-down – a txn_stepdown_ts_set 0 -> 1 or disagg_role_leader 1 -> 0 transition followed within 30s by a serverStatus.uptime reset.
      4. Cache pressure, step-down timestamp set vs unsetcache_bytes_inuse / cache_bytes_max, plotted separately for txn_stepdown_ts_set at 1 and at 0, as both an average and a P99.
      5. Stable not advancing to the step-down timestamptxn_global_stable_timestamp restricted to when txn_stepdown_ts_set is 1, plus how long txn_stepdown_ts_set stays at 1 as the wait duration.
      6. Rolled-back transactionstxn_rollback_stepdown. It only ever increases, so a 24h increase() grouped by cluster gives the "which clusters had the most rollbacks" view directly.

      Rewards

      1. Nodes successfully stepping down, with a total for the success percentage – successes from disagg_role_leader 1 -> 0 transitions, total from txn_stepdown_ts_set 0 -> 1 transitions, percentage as the ratio.
      2. Read performance remaining highlayered_curs_search, layered_curs_search_near, layered_curs_next, layered_curs_prev. Should stay flat across the whole period.
      3. Write performance high until reconfigurelayered_curs_insert, layered_curs_update, layered_curs_remove, layered_curs_modify. Expect flat until the reconfigure, then zero; a dip before the reconfigure is the signal to alert on.
      4. Schema operations while the step-down timestamp is setsession_table_create_success and session_table_create_fail, session_table_drop_success and session_table_drop_fail, session_table_publish_success and session_table_publish_fail, restricted to when txn_stepdown_ts_set is 1.

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Sid Mahajan
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: