ExportXMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Execution
    • QI 2025-02-17
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Overview

      The router-side (mongos) end-of-transaction (EOT) change stream synthesis is being built in PR 65802 (SERVER-137669, stage skeleton, inert until shards produce accounting events) and the sharded write path is still blocked on approvals. Metric definitions depend on neither. Add all EOT metrics now — registered and visible in serverStatus at their default values — plus first-side wiring for everything already populateable and unit-testable today: the mongod event counter, the router-stage metrics the PR's stage skeleton can exercise with synthetic control events, and the allowEndOfTransaction parameter gauge.

      Branch strategy: implement on top of the PR 65802 head branch (jan.steemann/SERVER-137669), rebase onto master once it merges.

      Semantics follow the Observability section of the EOT design doc (SPM-4826), with three agreed deviations:

      • rebuilds (total) + rebuildsFailed become rebuilds.succeeded + rebuilds.failed; totals are inferred by summing.
      • rebuildTimeMicros is a single OTEL histogram (count + cumulative sum + buckets), recording successful rebuilds only. 6 buckets (MaaS cap is 10): 1ms, 10ms, 100ms, 1s, 10s, 100s.
      • completedWithRemovedShards keeps its name under endOfTransaction.* so event.endOfTransaction stays a clean member of the future event.<operationType> family (SERVER-135657).

      Metric inventory

      All paths relative to serverStatus.metrics.changeStreams. ClusterRole controls node-type visibility.

      Router-only (ClusterRole::RouterServer):

      Path Instrument Semantics
      endOfTransaction.rebuilds.succeeded Counter On-demand rebuilds triggered by the lookback kickback that completed the resume
      endOfTransaction.rebuilds.failed Counter Rebuilds that reached the rebuild cap and failed the stream
      endOfTransaction.rebuildTimeMicros Histogram (kBucketCounts) Latency of successful rebuilds: kickback catch → resumed at consumer position
      endOfTransaction.synthesis.transactions.evicted Counter Tracked transactions abandoned after the eviction delay (whenAvailable disposition)
      endOfTransaction.synthesis.pbrtBumps Counter Explicit client-PBRT advances to the emitted EOT token
      endOfTransaction.synthesis.transactions.tracked UpDownCounter Current router-side transaction state entries, summed over open streams
      endOfTransaction.completedWithRemovedShards Counter EOT events emitted best-effort after a participant shard was removed; subset of event.endOfTransaction
      endOfTransaction.lookbackDistanceExceeded Counter Lookbacks refused beyond the maximum lookback distance

      mongod-only (ClusterRole::ShardServer):

      Path Instrument Semantics
      endOfTransaction.requiredMissingMetadata Counter Required streams failing on a commit entry without EOT metadata (committed while the flag/param was disabled)

      Both mongod and mongos (ClusterRole::None):

      Path Instrument Semantics
      event.endOfTransaction Counter EOT events emitted on this node, counted at synthesis time (mongod: applyOps synthesis; mongos: synthesized events). Future member of the event.<operationType> family (SERVER-135657)
      endOfTransaction.allowEndOfTransaction Gauge 0/1 Whether the allowEndOfTransaction server parameter is currently enabled

      Design

      No recorder class (KISS/YAGNI): recorded from six scattered sites with one process-global cell per metric and no polymorphism. Cell accessors are flat free functions in mongo::change_stream in change_stream_metrics_util.h (static-local pattern, like the existing error counters); call sites bump cells directly. One shared helper — the recordChangeEventEmitted(StringData) per-operationType dispatcher — lives centrally because it has two unrelated call files. The rebuild pairing rule (rebuilds.succeeded + rebuildTimeMicros must move together) is documented on the accessors; no helper until the kickback handler exists.

      Scope of Work

      • src/mongo/otel/metrics/metric_names.h — 11 new MetricNames constants
      • src/mongo/db/change_stream_metrics_util.h — 11 flat metric accessors + the dispatcher
      • src/mongo/db/pipeline/change_stream_unwind_transaction_stage.cpp — recordChangeEventEmitted in TransactionOpIterator::_createEndOfTransactionIfNeeded() (mongod synthesis birth site)
      • PR 65802's router stage files — recordChangeEventEmitted at synthesis in _handleAccountingControlEvent; trackedTransactions ± in EndOfTransactionAggregator::tryCompleteTransaction; pbrtBumps in the production PBRT-advancer path
      • src/mongo/db/end_of_transaction_parameter.idl — add on_update: callback refreshing the gauge (ftdc_server.idl precedent)

      Acceptance Criteria

      • All 11 metrics visible in serverStatus at default values, with correct node-type visibility (router-only absent on mongod, mongod-only absent on mongos, both-role present on both)
      • Router stage unit tests with synthetic accounting control events (existing _forTest hooks, OtelMetricsCapturer): trackedTransactions up/down through partial→completing events; event.endOfTransaction bumps at synthesis even when the user pipeline filters the event; pbrtBumps bumps only for events that survive the user pipeline and take the production advancer path
      • Mongod unit tests: unwind-stage synthesis bumps event.endOfTransaction; dispatcher dispatches endOfTransaction and ignores unknown types; gauge flips with gAllowEndOfTransaction
      • Remaining metrics stay at 0 until the follow-up machinery lands (recording-site map in the spec is the contract)

      Out of Scope

      • e2e/jstest coverage — deferred until endOfTransaction events are emitted under featureFlagEndOfTransactionChangeEventSynthesis (requires the write-path accounting events); unit tests cover the metric plumbing today
      • Population of rebuilds.succeeded/failed, rebuildTimeMicros, synthesis.transactions.evicted, completedWithRemovedShards, lookbackDistanceExceeded, requiredMissingMetadata — kickback/eviction/shard-availability machinery does not exist yet
      • Real accounting control event production (the sharded write path work)
      • option.showEndOfTransaction.* counters — already implemented
      • The generic per-operationType event counter expansion (SERVER-135657) — this ticket only establishes the dispatcher seam

            Assignee:
            Denis Grebennicov
            Reporter:
            Denis Grebennicov
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: