In change stream queries, it is common to see long-running oplog scans, because change streams can be opened for a point in the past. A problem occurs when the change stream's filter does not match any or only few oplog entries during a long scan. In this case no intermediate results will be returned by the change stream, and no updated PBRT will be returned to the consumer. The consumer has no way to determine if the change stream is actually making progress or not. If the scan fails for whatever reason, the consumer can only resume the scan from the very beginning, as no updated PBRTs were returned from the scan.

The changes in this PR add a query knob that allows to periodically interrupt lengthy oplog scans in change streams, so that they return the so-far accumulated results even if the scan has not reached the end of the oplog or has filled up an entire response batch. This allows consumers to observe progress of the change stream, and receive updated PBRTs so that in case of error they can resume from the last known position of the scan.

This query knob is named internalOperationResponseMaxMS, and it can be set via setParameter on mongod nodes. The query knob value can be overridden for individual change stream queries as well, by passing it in the change stream's initial aggregate command via query settings, e.g.

{querySettings: {queryKnobs:

}}

This is supported starting with FCV 9.0.
The default value for the query knob internalOperationResponseMaxMS is 0, meaning that change stream oplog scans will not be time-limited by default and will not return updated PBRTs during the scan. This preserves the original behavior from previous versions for backwards-compatibility, and make the new functionality strictly opt-in.

Overriding the default value of the internalOperationResponseMaxMS query knob to a value > 0 will make change stream oplog scans periodically check if the configured time limit is exceeded, and if so, they will return the so-far accumulated results back with an updated PBRT.

Important caveat: the name of the query knob is internalOperationResponseMaxMS, which suggests broad applicability across all/many types of queries, but the knob currently only has an effect on change stream queries! Setting it on other types of queries will have no effect.

Description of Linked Ticket

In versions before 9.0, the postBatchResumeToken (PBRT) that is returned in responses to change stream getMore requests is currently not updated if the getMore responses do not include any documents {}and{} the change stream cursor has not yet reached the oplog tail.
In this case, the getMore responses always return the same PBRT value, before either a matching document is returned or the change stream cursor is at the end of the oplog and the high water mark gets updated.

Not returning an updated an updated PBRT is problematic in case when there is a large backlog of oplog entries to scan by the change stream cursor, there is a match expression that filters out all/most events and there is only a limited amount of time available before the stream gets invalidated.
For example, we found such case in an internal test that executes stepdowns of the primary every 8 seconds, and in which getting to the tail of the oplog with a single change stream cursor takes longer than 8 seconds. The slow scanning here is likely due to the test running under UBSAN, but in theory the problem can also happen without it.
Once a stepdown occurs, the stream is aborted. The test then retries and reopens a new change stream cursor at the same operation start time as the previous one. As none of the getMore requests returns an updated PBRT, the test has no chance to make incremental progress: it always needs to restart at the same oplog position and will never manage to get to the end of the oplog within the available 8 seconds.

This ticket adds infrastructure for bailing out of such lengthy oplog scans every now and then, without failing the entire change stream query. This allows the consumer of the change stream to take note of updated PBRTs and the change stream making progress, even while the oplog scan is still ongoing. A frequently updated resume token also allows consumers to resume the scan from a later point, instead of having to resume the scan from the very beginning in case no progress gets reported intermittently.

The solution taken in this change is to add an opt-in query knob that allows to periodically interrupt lengthy collection scans in change stream, so that they return the so-far accumulated results even if they have not reached the end of the oplog or have filled up an entire response batch.

This query knob is named internalOperationResponseMaxMS and it can be set via setParameter on mongod nodes. The query knob value can be overridden for individual change stream queries as well, by passing it in the change stream's initial aggregate command via query settings, e.g.

{querySettings: {queryKnobs: {operationResponseMaxMS: <value>}}} 

The default value for the query knob is 0, meaning that change stream oplog scans will not be time-limited by default. That makes the feature strictly opt-in.

Overriding the default value to a value > 0 will make change stream oplog scans periodically check if the configured time limit is exceeded, and if so, they will return the so-far accumulated results back with an updated PBRT.

Important caveat: the name of the query knob internalOperationResponseMaxMS suggests broad applicability across all/many types of queries, but the knob currently only has an effect on change stream queries!