In versions before 9.0, the postBatchResumeToken (PBRT) that is returned in responses to change stream getMore requests is currently not updated if the getMore responses do not include any documents {}and{} the change stream cursor has not yet reached the oplog tail.
In this case, the getMore responses always return the same PBRT value, before either a matching document is returned or the change stream cursor is at the end of the oplog and the high water mark gets updated.

Not returning an updated an updated PBRT is problematic in case when there is a large backlog of oplog entries to scan by the change stream cursor, there is a match expression that filters out all/most events and there is only a limited amount of time available before the stream gets invalidated.
For example, we found such case in an internal test that executes stepdowns of the primary every 8 seconds, and in which getting to the tail of the oplog with a single change stream cursor takes longer than 8 seconds. The slow scanning here is likely due to the test running under UBSAN, but in theory the problem can also happen without it.
Once a stepdown occurs, the stream is aborted. The test then retries and reopens a new change stream cursor at the same operation start time as the previous one. As none of the getMore requests returns an updated PBRT, the test has no chance to make incremental progress: it always needs to restart at the same oplog position and will never manage to get to the end of the oplog within the available 8 seconds.

This ticket adds infrastructure for bailing out of such lengthy oplog scans every now and then, without failing the entire change stream query. This allows the consumer of the change stream to take note of updated PBRTs and the change stream making progress, even while the oplog scan is still ongoing. A frequently updated resume token also allows consumers to resume the scan from a later point, instead of having to resume the scan from the very beginning in case no progress gets reported intermittently.

The solution taken in this change is to add an opt-in query knob that allows to periodically interrupt lengthy collection scans in change stream, so that they return the so-far accumulated results even if they have not reached the end of the oplog or have filled up an entire response batch.

This query knob is named internalOperationResponseMaxMS and it can be set via setParameter on mongod nodes. The query knob value can be overridden for individual change stream queries as well, by passing it in the change stream's initial aggregate command via query settings, e.g.

{querySettings: {queryKnobs: {operationResponseMaxMS: <value>}}} 

Setting the knob value for individual queries allows consumers to opt-in on a per-query basis, which is likely preferrable over the coarse-grained approach of setting the server parameter for every change stream query.

Setting the knob value via query settings builds on the functionality introduced in 9.0 by SPM-4364, and requires FCV 9.0.

The default value for the query knob internalOperationResponseMaxMS is 0, meaning that change stream oplog scans will not be time-limited by default and will not return updated PBRTs during the scan. This preserves the original behavior from previous versions for backwards-compatibility, and make the new functionality strictly opt-in.

Overriding the default value of the internalOperationResponseMaxMS query knob to a value > 0 will make change stream oplog scans periodically check if the configured time limit is exceeded, and if so, they will return the so-far accumulated results back with an updated PBRT.

Important caveat: the name of the query knob is internalOperationResponseMaxMS, which suggests broad applicability across all/many types of queries, but the knob currently only has an effect on change stream queries! Setting it on other types of queries will currently have no effect. The functionality may be extended in the future, but right now it only applies to change stream queries.

The main purpose of the internalOperationResponseMaxMS knob is to bound the time that change stream aggregate / getMore commands can spend on a lengthy oplog scan. Without setting the knob value, the maximum time for such commands is unbounded. When using the knob, the time spent in such operations will be bounded. Typically, the knob value should be set to values larger than what's used for maxTimeMS / maxAwaitTimeMS so that change streams still make substantial progress in oplog scanning on every command invocation. Setting a too-low value for internalOperationResponseMaxMS will make the scans bail out quickly, but will limit the progress that an oplog scan can make on each command invocation. If the oplog scans are interrupted to frequently because of a too-low knob value, this can make the oplog scans fall behind and not being able to catch up with the write workload.
As a guideline, the internalOperationResponseMaxMS value should be set to a value substantially larger than maxTimeMS / maxAwaitTimeMS, but to a value that is lower than the network timeout value. This way the aggregate / getMore commands can each make substantial progress on the oplog scan, while still returning updates eventually and before the connection times out.