-
Type:
Improvement
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Query Execution
-
QI 2025-02-17
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Overview
The router-side (mongos) end-of-transaction (EOT) change stream synthesis is being built in PR 65802 (SERVER-137669, stage skeleton, inert until shards produce accounting events) and the sharded write path is still blocked on approvals. Metric definitions depend on neither. Add all EOT metrics now — registered and visible in serverStatus at their default values — plus first-side wiring for everything already populateable and unit-testable today: the mongod event counter, the router-stage metrics the PR's stage skeleton can exercise with synthetic control events, and the allowEndOfTransaction parameter gauge.
Branch strategy: implement on top of the PR 65802 head branch (jan.steemann/SERVER-137669), rebase onto master once it merges.
Semantics follow the Observability section of the EOT design doc (SPM-4826), with three agreed deviations:
- rebuilds (total) + rebuildsFailed become rebuilds.succeeded + rebuilds.failed; totals are inferred by summing.
- rebuildTimeMicros is a single OTEL histogram (count + cumulative sum + buckets), recording successful rebuilds only. 6 buckets (MaaS cap is 10): 1ms, 10ms, 100ms, 1s, 10s, 100s.
- completedWithRemovedShards keeps its name under endOfTransaction.* so event.endOfTransaction stays a clean member of the future event.<operationType> family (SERVER-135657).
Metric inventory
All paths relative to serverStatus.metrics.changeStreams. ClusterRole controls node-type visibility.
Router-only (ClusterRole::RouterServer):
| Path | Instrument | Semantics |
|---|---|---|
| endOfTransaction.rebuilds.succeeded | Counter | On-demand rebuilds triggered by the lookback kickback that completed the resume |
| endOfTransaction.rebuilds.failed | Counter | Rebuilds that reached the rebuild cap and failed the stream |
| endOfTransaction.rebuildTimeMicros | Histogram (kBucketCounts) | Latency of successful rebuilds: kickback catch → resumed at consumer position |
| endOfTransaction.synthesis.transactions.evicted | Counter | Tracked transactions abandoned after the eviction delay (whenAvailable disposition) |
| endOfTransaction.synthesis.pbrtBumps | Counter | Explicit client-PBRT advances to the emitted EOT token |
| endOfTransaction.synthesis.transactions.tracked | UpDownCounter | Current router-side transaction state entries, summed over open streams |
| endOfTransaction.completedWithRemovedShards | Counter | EOT events emitted best-effort after a participant shard was removed; subset of event.endOfTransaction |
| endOfTransaction.lookbackDistanceExceeded | Counter | Lookbacks refused beyond the maximum lookback distance |
mongod-only (ClusterRole::ShardServer):
| Path | Instrument | Semantics |
|---|---|---|
| endOfTransaction.requiredMissingMetadata | Counter | Required streams failing on a commit entry without EOT metadata (committed while the flag/param was disabled) |
Both mongod and mongos (ClusterRole::None):
| Path | Instrument | Semantics |
|---|---|---|
| event.endOfTransaction | Counter | EOT events emitted on this node, counted at synthesis time (mongod: applyOps synthesis; mongos: synthesized events). Future member of the event.<operationType> family (SERVER-135657) |
| endOfTransaction.allowEndOfTransaction | Gauge 0/1 | Whether the allowEndOfTransaction server parameter is currently enabled |
Design
No recorder class (KISS/YAGNI): recorded from six scattered sites with one process-global cell per metric and no polymorphism. Cell accessors are flat free functions in mongo::change_stream in change_stream_metrics_util.h (static-local pattern, like the existing error counters); call sites bump cells directly. One shared helper — the recordChangeEventEmitted(StringData) per-operationType dispatcher — lives centrally because it has two unrelated call files. The rebuild pairing rule (rebuilds.succeeded + rebuildTimeMicros must move together) is documented on the accessors; no helper until the kickback handler exists.
Scope of Work
- src/mongo/otel/metrics/metric_names.h — 11 new MetricNames constants
- src/mongo/db/change_stream_metrics_util.h — 11 flat metric accessors + the dispatcher
- src/mongo/db/pipeline/change_stream_unwind_transaction_stage.cpp — recordChangeEventEmitted in TransactionOpIterator::_createEndOfTransactionIfNeeded() (mongod synthesis birth site)
- PR 65802's router stage files — recordChangeEventEmitted at synthesis in _handleAccountingControlEvent; trackedTransactions ± in EndOfTransactionAggregator::tryCompleteTransaction; pbrtBumps in the production PBRT-advancer path
- src/mongo/db/end_of_transaction_parameter.idl — add on_update: callback refreshing the gauge (ftdc_server.idl precedent)
Acceptance Criteria
- All 11 metrics visible in serverStatus at default values, with correct node-type visibility (router-only absent on mongod, mongod-only absent on mongos, both-role present on both)
- Router stage unit tests with synthetic accounting control events (existing _forTest hooks, OtelMetricsCapturer): trackedTransactions up/down through partial→completing events; event.endOfTransaction bumps at synthesis even when the user pipeline filters the event; pbrtBumps bumps only for events that survive the user pipeline and take the production advancer path
- Mongod unit tests: unwind-stage synthesis bumps event.endOfTransaction; dispatcher dispatches endOfTransaction and ignores unknown types; gauge flips with gAllowEndOfTransaction
- Remaining metrics stay at 0 until the follow-up machinery lands (recording-site map in the spec is the contract)
Out of Scope
- e2e/jstest coverage — deferred until endOfTransaction events are emitted under featureFlagEndOfTransactionChangeEventSynthesis (requires the write-path accounting events); unit tests cover the metric plumbing today
- Population of rebuilds.succeeded/failed, rebuildTimeMicros, synthesis.transactions.evicted, completedWithRemovedShards, lookbackDistanceExceeded, requiredMissingMetadata — kickback/eviction/shard-availability machinery does not exist yet
- Real accounting control event production (the sharded write path work)
- option.showEndOfTransaction.* counters — already implemented
- The generic per-operationType event counter expansion (SERVER-135657) — this ticket only establishes the dispatcher seam
- is related to
-
SERVER-137669 Create router synthesis stage skeleton
-
- Closed
-
-
SERVER-135657 Add generic per-operationType counters for regular (non-control) change stream events
-
- Backlog
-