Downstream Changes Summary

Project context
---------------
Version-2 change streams (featureFlagChangeStreamReaderV2, SPM-1941) replace the
open-a-cursor-on-every-shard model with precise shard targeting: the router consults placement
history and opens cursors only on shards that hold data for the stream's scope (collection /
database / all-databases). Topology changes are absorbed by a router-side state machine
(ChangeStreamHandleTopologyChangeV2Stage) that re-targets the stream; while a stream's segment
lives on a removed shard, the stream runs in "degraded" mode until re-targeting completes. The
targeters also observe DDL/sharding "control events" (chunk moves, primary moves, namespace
placement changes, database creations) that can invalidate current targeting. All of this is
invisible in serverStatus today; this ticket makes each part observable.

What this ticket adds
---------------------
A brand-new serverStatus section, metrics.changeStreams.shardTargeting (router-side; also
exported through OpenTelemetry under the mongodb.serverStatus.metrics.* prefix). No new commands,
options, or wire-protocol changes; no behavioral change to targeting. Metric groups:

1. Topology-handler state machine (per v2 change stream on the router)

• topologyState.<state> counters - one ENTRY counter per state of the topology-change
  handler: waiting, fetchingInitialization, fetchingGettingChangeEvent,
  fetchingStartingChangeStreamSegment, fetchingNormalGettingChangeEvent,
  fetchingDegradedGettingChangeEvent, downgrading, final. Counts state transitions, so the
  mix shows where re-targeting time goes (e.g. fetchingDegraded* rising = streams stuck
  re-targeting after shard removal).

2. Degraded-mode gauge

• degraded (up/down counter, gauge): how many stream segments are CURRENTLY degraded (their
  shard was removed and re-targeting has not completed). +1 on entering degraded mode, -1 on
  recovery or teardown.

3. Targeter scope (per targeter construction/(re-)install)

• targeterScope.collection. {dbPresent,dbAbsent}, targeterScope.database.{dbPresent,dbAbsent}

  (counters): event-handler installations of collection-/database-scoped targeters, split by
  whether the underlying database was present at install time.

• targeterScope.allDatabases (counter): installations of the all-databases targeter (no
  presence split).

4. Control events

• controlEvents. {moveChunk,movePrimary,namespacePlacementChanged,databaseCreated}

  (counters):
  control events actually observed by a live v2 targeter - the raw driver volume behind
  re-targeting activity.

5. Placement-history lookups (config server round-trips)

• placementHistoryLookup. {ok,notAvailable,futureClusterTime}

  (counters): lookup outcomes -
  success, placement not (yet) available, and queried ahead of known placement time
  respectively. notAvailable/futureClusterTime spikes point at placement-history lag.

• placementHistoryLookup.latencyMillis (histogram, MILLISECONDS): wall time of each
  placement-history lookup. Note the unit is milliseconds here, unlike the updateLookup/
  idLookup latency histograms (microseconds).

Population: all counters record only when v2 change streams are actually in use (flag on,
streams running); the section appears zeroed from startup regardless. Values are per router
process. Backport to v9.0 requested (BACKPORT-30094).

Description of Linked Ticket

Problem

The V2 shard-targeting state machine (SPM-1941) is a 9-state FSM running on mongos, but we have no visibility into its runtime behavior. We currently expose cursor/throughput/error metrics, but nothing tells us whether the FSM itself is healthy, whether it's stuck in degraded mode, how much topology churn it's processing, or whether placement history lookups (which the FSM depends on) are fast. This matters for backporting to 9.0, where we'll want fast triage signals without needing to attach a debugger or reason from logs alone.

Scope of Work

1. Per-state transition counters

• Add a counter per state transition of the ChangeStreamHandleTopologyChangeV2Stage::State enum (src/mongo/db/exec/agg/change_stream_handle_topology_change_v2_stage.h, 9 values): kUninitialized, kWaiting, kFetchingInitialization, kFetchingGettingChangeEvent, kFetchingStartingChangeStreamSegment, kFetchingNormalGettingChangeEvent, kFetchingDegradedGettingChangeEvent, kDowngrading, kFinal
• Exposed via serverStatus or a dedicated section, scoped per change stream / mongos
• The targeter's scope dimension (Collection/Database/AllDatabases) is a separate concern, tracked as its own set of metrics rather than folded into these state-transition counters

2. Degraded mode counter

• Add a degraded counter that increments when a stream enters degraded mode (bounded segments for removed shards, undoNextReady() boundary handling)
• Should make it trivial to tell "is any active stream currently degraded" at a glance

3. Control event counters

• Per-type counters on mongos for the 4 control event types: moveChunk, movePrimary, namespacePlacementChanged, databaseCreated
• Purpose: surface topology churn volume and confirm the control event pipeline is actually flowing (a stuck pipeline should be visible as a stalled counter, not silence)

4. Placement history query metrics

• Add counter + latency histogram for placement history lookups, bucketed by result status: ok, futureClusterTime, notAvailable
• Follow the same pattern already used for the updateLookup executor metrics, for consistency

5. (Exploratory) AsyncResultsMerger metrics

• src/mongo/s/query/exec/async_results_merger.cpp is where per-shard cursors are merged into the mongos-side stream, so it may be a good place for per-remote-cursor getMore latency/error metrics
• Not fully scoped yet — needs investigation into what AsyncResultsMerger already tracks (it has a query_stats::DataBearingNodeMetrics member) before deciding what, if anything, to add here

Acceptance Criteria

• State transition counters visible per ChangeStreamHandleTopologyChangeV2Stage::State value
• degraded counter increments/decrements (or is a gauge) correctly on entry/exit from degraded mode
• Control event counters increment per event type on mongos
• Placement history metric emitted on every query with correct status bucket and reasonable latency bucket boundaries
• AsyncResultsMerger investigation completed and a decision made on whether to add metrics there
• All new metrics covered by unit/integration tests
• No measurable overhead on the hot cursor path

Related Files

• src/mongo/db/exec/agg/change_stream_handle_topology_change_v2_stage.h — V2 topology-change-handling state machine (State enum, L42-74)
• src/mongo/s/change_streams/ — targeter scopes and placement history query path
• src/mongo/s/query/exec/async_results_merger.cpp — candidate location for per-remote-cursor metrics (exploratory)
• Reference pattern: updateLookup executor metrics (see cs-local-lookup-eligibility work, SERVER-129058/129059)