Adjust behavior for emitting additional fields in change streams

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fixed
    • Priority: Major - P3
    • 8.3.0-rc0
    • Affects Version/s: 8.3 Required, 8.2.1
    • Component/s: Change streams
    • None
    • Query Execution
    • Minor Change
    • ALL
    • v8.2
    • QE 2025-09-01
    • None
    • 3
    • TBD
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Adjust the behavior for emitting additional fields in change streams after SERVER-86688 caused a performance regression for change stream consumers in v8.2.0, and HELP-80004 was opened because of that.

      We discussed several mitigation options in this document, and decided to go with option 2b.

      This solution will introduce a new internal boolean flag, showCommitTimestamp to surface the commitTimestamp field. The flag defaults to false, i.e. the commitStamp field will not be emitted by default. That also means the commitTimestamp field will not cause any performance overhead unless consumers opt-in to receiving it. We will not document/publish the new internal flag. We plan to only use it internally in the resharding change streams monitor.

      The collectionUUID and updateDescription.disambiguatedPaths will again be hidden behind the already existing showExpandedEvents flag. This restores the pre-v8.2.0 behavior for emitting these fields. If the flag is not set, there will be no performance overhead for these fields. If the flag is set, the performance overhead will be the same as in pre-v8.2.0 versions.

      With this option implemented, no code changes are needed on the Search side, except that they potentially will not make use of their field-filtering mechanism anymore. This is up to the Search team to decide.

      For external change streams users, this change will restore full compatibility in behavior and performance with pre-v8.2.0 versions.
      The solution is incompatible with the behavior in (just) v8.2.0, because v8.2.0 exposed the three fields unconditionally. We decided that this incompatibility is acceptable, and it is more important to get rid of the performance regression that was caused by SERVER-86688.

      We need to ensure that after the change is made, the resharding change streams monitor must set the showCommitTimestamp field when opening a change stream.

              Assignee:
              Jan Steemann
              Reporter:
              Jan Steemann
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: