This ticket is a feature request.

SERVER-102900 recently added a new query knob internalOperationResponseMaxMS which can be set on a node globally via setParameter or for individual queries via query settings (then named operationResponseMaxMS).

The current behavior of the knob is that it limits the amount of time that every change stream aggregate / getMore command spends during oplog processing. For change streams, it is quite common to open the change stream at a timepoint that is way in the past, which can require a large part of the oplog to be scanned. If during oplog scanning no results are returned (e.g. because the scanned oplog entries do not match the change stream's match expression) then the scan may take an exorbitant amount of time.
Before addition of the knob, there was no way for change stream consumers to interrupt such lengthy oplog scan and return intermittent results (this may be an empty result, but it will be accompanied by an updated PBRT that reflects the progress of the scan). The knob from SERVER-102900 now allows that.

However, the knob is currently only effective on change stream queries. It has no effect for any other queries.

The request in this ticket is that the scope of the knob should be extended so that in the future it can also be used to interrupt other, non-change stream queries, bailing out of lengthy, otherwise blocking operations, without failing the entire query. For example, a long-running blocking sort operation or a full collection scan with a filter that does not produce any matches may take an arbitrary amount of time before completing or returning first results.

It would be great if for such long-running queries we could also limit the time that is spent in every request, and return intermediate results to report the current progress.
The idea is that when the knob is used and the time limit is hit, the command returns the already accumulated results (if any - maybe an empty batch) and returns control to the caller. The caller can then send further getMore requests to the query and continue the operation. This has the advantage that requests made by the caller do not hang "indefinitely" without making any apparent progress.
With the knob set, callers will get responses back intermittendly. Every response will return updated metrics, so that the caller can better observe the progress of the operation.
Also, this avoids that long-running requests run into socket timeouts, as long-running requests will be avoided with the knob.

The implementation in SERVER-102900 provides that, but currently only for change stream queries. To make it usable for additional types of queries, we need to the following:

• validate that returning an empty intermittent result from a query (because the time limit hits) does not prevent the query from being continued later. This could be problematic if callers interpret receiving an empty result as "end of query" in regular queries.
• if this turns out to be ok, we need to broaden the scope of the knob in plan_executor_impl.cpp here, where it is currently restricted to just change stream queries. This may make the knob work for other queries in the classic engine, but the full effect needs to be evaluated.
• check if the knob should be supported for write operations.
• implement the same functionality for the SBE engine. SBE currently has no support for this knob.
• validate how queries with sub-pipelines (e.g. $lookup, $graphLookup, $unionWith) behave when setting the knob, and if the functionality needs to be added to stages containing sub-pipelines. This is currently unclear.
• check if the express executor should support the functionality as well. Potentially it is not worth to add support because the types of queries it executes are mostly short-running anyway.

Overall, this will require some initial investigation and feasibility analysis.