ExportXMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Storage Engines, Storage Engines - Transactions
    • 194.769
    • SE Persistence backlog
    • None

      Background

      Design: WRITING-39975. Epic: SPM-4886.

      Change

      Build a Grafana dashboard to watch the safety of clusters with the feature on, and run a staged rollout: internal dev clusters, then production clusters in widening steps. Each step runs long enough to include step-ups. Move on only when the dashboard is clean. If it shows a problem, turn the kill switch off.

      Metrics:

      • layered_curs_blind_remove_follower rate per node: above zero on every follower where the feature is on.
      • Orphan removes per cluster (layered_curs_blind_remove_ingest_deleted + disagg_step_up_ingest_drain_orphan_remove): near zero, alert when rising.
      • layered_curs_blind_remove_leader: zero.
      • Validation failures and index errors: zero, alert on any.
      • Server asserts, crashes and restarts: no increase compared with clusters where the feature is off.

      Acceptance criteria

      • Dashboard and alerts are live. Each rollout step is recorded as clean before the next.

      Depends on

      MongoDB testing ticket (proctor override removal).

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Chenhao Qu
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: