-
Type:
Improvement
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: None
-
None
-
DB Integration & Observability
-
Fully Compatible
-
v9.0
-
QI 2025-02-17
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Problem
The V2 shard-targeting state machine (SPM-1941) is a 9-state FSM running on mongos, but we have no visibility into its runtime behavior. We currently expose cursor/throughput/error metrics, but nothing tells us whether the FSM itself is healthy, whether it's stuck in degraded mode, how much topology churn it's processing, or whether placement history lookups (which the FSM depends on) are fast. This matters for backporting to 9.0, where we'll want fast triage signals without needing to attach a debugger or reason from logs alone.
Scope of Work
1. Per-state transition counters
- Add a counter per state transition of the ChangeStreamHandleTopologyChangeV2Stage::State enum (src/mongo/db/exec/agg/change_stream_handle_topology_change_v2_stage.h, 9 values): kUninitialized, kWaiting, kFetchingInitialization, kFetchingGettingChangeEvent, kFetchingStartingChangeStreamSegment, kFetchingNormalGettingChangeEvent, kFetchingDegradedGettingChangeEvent, kDowngrading, kFinal
- Exposed via serverStatus or a dedicated section, scoped per change stream / mongos
- The targeter's scope dimension (Collection/Database/AllDatabases) is a separate concern, tracked as its own set of metrics rather than folded into these state-transition counters
2. Degraded mode counter
- Add a degraded counter that increments when a stream enters degraded mode (bounded segments for removed shards, undoNextReady() boundary handling)
- Should make it trivial to tell "is any active stream currently degraded" at a glance
3. Control event counters
- Per-type counters on mongos for the 4 control event types: moveChunk, movePrimary, namespacePlacementChanged, databaseCreated
- Purpose: surface topology churn volume and confirm the control event pipeline is actually flowing (a stuck pipeline should be visible as a stalled counter, not silence)
4. Placement history query metrics
- Add counter + latency histogram for placement history lookups, bucketed by result status: ok, futureClusterTime, notAvailable
- Follow the same pattern already used for the updateLookup executor metrics, for consistency
5. (Exploratory) AsyncResultsMerger metrics
- src/mongo/s/query/exec/async_results_merger.cpp is where per-shard cursors are merged into the mongos-side stream, so it may be a good place for per-remote-cursor getMore latency/error metrics
- Not fully scoped yet — needs investigation into what AsyncResultsMerger already tracks (it has a query_stats::DataBearingNodeMetrics member) before deciding what, if anything, to add here
Acceptance Criteria
- State transition counters visible per ChangeStreamHandleTopologyChangeV2Stage::State value
- degraded counter increments/decrements (or is a gauge) correctly on entry/exit from degraded mode
- Control event counters increment per event type on mongos
- Placement history metric emitted on every query with correct status bucket and reasonable latency bucket boundaries
- AsyncResultsMerger investigation completed and a decision made on whether to add metrics there
- All new metrics covered by unit/integration tests
- No measurable overhead on the hot cursor path
Related Files
- src/mongo/db/exec/agg/change_stream_handle_topology_change_v2_stage.h — V2 topology-change-handling state machine (State enum, L42-74)
- src/mongo/s/change_streams/ — targeter scopes and placement history query path
- src/mongo/s/query/exec/async_results_merger.cpp — candidate location for per-remote-cursor metrics (exploratory)
- Reference pattern: updateLookup executor metrics (see cs-local-lookup-eligibility work,
SERVER-129058/129059)
- is depended on by
-
COMPASS-11244 Investigate changes in SERVER-135656: Add metrics for change streams shard-targeting V2 state machine, control events, and placement history
-
- Closed
-
-
TOOLS-4409 Investigate changes in SERVER-135656: Add metrics for change streams shard-targeting V2 state machine, control events, and placement history
-
- Closed
-
- is related to
-
SERVER-129058 Implement ShardedClusterLocalLookupEligibility for sharded change-stream updateLookup
-
- Closed
-
- related to
-
SERVER-135657 Add generic per-operationType counters for regular (non-control) change stream events
-
- Backlog
-