SERVER-86688 previously changed which fields would be emitted as part of certain change stream DML events (insert, update, replace, delete). Before SERVER-86688, the fields collectionUUID and updateDescription.disambiguatedPaths (only present for update change events) were only emitted if the change stream was opened with the showExpandedEvents flag.

SERVER-86688 (merged into master and present in v8.2.0) will emit these fields unconditionally, regardless of the value of showExpandedEvents.

Emitting the fields unconditionally led to larger change events being returned and caused a performance regression.
This change (SERVER-109626) reverts the changes from SERVER-86688, meaning that the two fields mentioned above will now again be emitted only if the change stream is opened with the showExpandedEvents flag.
The change will also be backported to the v8.2 branch and will be included in v8.2.1 and higher.

The change is downwards-incompatible to v8.2.0, because in v8.2.0 the two fields are emitted unconditionally, whereas all versions that include this fix will emit them only if showExpandedEvents is set.

This can be problematic when upgrading from (exactly) v8.2.0 to a higher version if change stream consumers opened the change stream without the showExpandedEvents flag and implicitly relied on the two fields being present regardless. Upgrading to v8.2.1 or higher will make these fields disappear unless showExpandedEvents is used again.

To summarize the behavior in different versions:

• versions < v8.2.0: collectionUUID and updateDescription.disambiguatedPaths are only emitted if the change stream is opened with the showExpandedEvents flag
• v8.2.0: collectionUUID and updateDescription.disambiguatedPaths are emitted unconditionally
• versions >= v8.2.1: collectionUUID and updateDescription.disambiguatedPaths are only emitted if the change stream is opened with the showExpandedEvents flag

This change also leads to the commitTimestamp field (only present for events that are part of a prepared transaction) not being emitted unless the change stream is opened with the internal flag showCommitTimestamp. The commitTimestamp field was introduced in v8.2.0 for events that are part of a prepared transaction and was emitted unconditionally for these. It was not present in versions before v8.2.0 at all. The commit timestamp field is used internally be the resharding, but probably has no larger use cases outside of that.
We decided that the showCommitTimestamp flag is an internal flag and should not be documented publicly. However, internal components and our tools can make use of it if required.

To summarize the behavior w.r.t. the commitTimestamp field in different versions:

• versions < v8.2.0: commitTimestamp field does not exist
• v8.2.0: commitTimestamp field will be emitted for all events that are part of prepared transactions
• versions >= v8.2.1: commitTimestamp is only emitted if the change stream is opened with the internal showCommitTimestamp flag

  Description of Linked Ticket

  Adjust the behavior for emitting additional fields in change streams after SERVER-86688 caused a performance regression for change stream consumers in v8.2.0, and HELP-80004 was opened because of that.

We discussed several mitigation options in this document, and decided to go with option 2b.

This solution will introduce a new internal boolean flag, showCommitTimestamp to surface the commitTimestamp field. The flag defaults to false, i.e. the commitStamp field will not be emitted by default. That also means the commitTimestamp field will not cause any performance overhead unless consumers opt-in to receiving it. We will not document/publish the new internal flag. We plan to only use it internally in the resharding change streams monitor.

The collectionUUID and updateDescription.disambiguatedPaths will again be hidden behind the already existing showExpandedEvents flag. This restores the pre-v8.2.0 behavior for emitting these fields. If the flag is not set, there will be no performance overhead for these fields. If the flag is set, the performance overhead will be the same as in pre-v8.2.0 versions.

With this option implemented, no code changes are needed on the Search side, except that they potentially will not make use of their field-filtering mechanism anymore. This is up to the Search team to decide.

For external change streams users, this change will restore full compatibility in behavior and performance with pre-v8.2.0 versions.
The solution is incompatible with the behavior in (just) v8.2.0, because v8.2.0 exposed the three fields unconditionally. We decided that this incompatibility is acceptable, and it is more important to get rid of the performance regression that was caused by SERVER-86688.

We need to ensure that after the change is made, the resharding change streams monitor must set the showCommitTimestamp field when opening a change stream.