ExportXMLWordPrintableJSON

    • Type: Task
    • Resolution: Fixed
    • Priority: Major - P3
    • 9.1.0-rc0, 9.1.0-rc1026
    • Affects Version/s: None
    • Component/s: None
    • None
    • Fully Compatible
    • QE 2026-11-23
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Summary

      Add two router-side server parameters that bound on-demand lookback ("backfill") for endOfTransaction change events, per the design doc "Emit End-of-Transaction Events with Shard Targeting Support" (sections "Maximum lookback distance" and "Rebuild cap"). Both are settable globally and overridable per change stream via query settings.

      Parameters

      Server parameter Query settings wire name Bounds
      internalChangeStreamEndOfTransactionMaxLookbackSeconds changeStreamEndOfTransactionMaxLookbackSeconds Length
      of a single lookback: the oplog timestamp distance (T0 - F) in seconds between the resume point T0
      and the lookback floor F (commitFloor).
      internalChangeStreamEndOfTransactionMaxRebuilds changeStreamEndOfTransactionMaxRebuilds Total
      number of kickback-driven rebuilds over a stream's lifetime.

      Define both in src/mongo/db/query/query_execution_knobs.idl as query-settings-settable, following
      the pattern of existing query knobs. Default values and validation ranges are not specified in the
      design (the rebuild cap default is "settled during implementation"); pick them here, with a positive
      lower bound, and document the choice. Both values must be representable in query settings.

      Behavior

      Max lookback distance

      • When the synthesis stage raises ChangeStreamEndOfTransactionBackfillRequired with floor F for a
        stream whose resume point is T0, the execution layer checks T0 - F against the configured value.
        If it exceeds the value, do not rebuild.
      • required streams: fail fast with a descriptive, non-resumable fatal error naming the lookback
        distance and the configured bound. The error code is decided in this ticket (reuse
        ChangeStreamHistoryLost or add a dedicated code).
      • whenAvailable streams: do not fail. The transactions that needed the refused lookback are
        abandoned and handled by the aggregator's eviction disposition (omitted, logged, counted).
      • The stage records a refused floor and does not re-raise the kickback for any floor >= a previously
        refused floor, so a refused lookback is not retried on every pass.
      • Increment metrics.changeStreams.endOfTransaction.lookbackDistanceExceeded.

      Rebuild cap

      • Each rebuild decrements the stream's remaining budget. When the cap is reached, the execution
        layer stops catching the kickback and fails the stream with a descriptive fatal error. It must not
        loop.
      • The cap is a backstop for the one-round-per-rebuild invariant and the bound on total rebuilds. It
        does not bound an individual lookback (that is the max lookback distance).
      • Increment metrics.changeStreams.endOfTransaction.rebuilds on each rebuild. Reaching the cap should
        also be recorded by a metric.

            Assignee:
            Lyublena Antova
            Reporter:
            Lyublena Antova
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: