Unify the total-vs-remaining conventions for forwarding maxTimeMS on shard-bound query commands

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Minor - P4
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Execution
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Overview

      Routers forward maxTimeMS to shards using two different and inconsistent conventions. Pick one and apply it to every shard-bound query command.

      Background

      When a router dispatches a query command to a shard, it sets the outgoing command's maxTimeMS.
      Two conventions coexist today:

      • Resolved total budget - the operation's whole budget, resolved from query settings via query_settings::resolveMaxTimeMSForShardForwarding():
        • src/mongo/s/query/planner/cluster_find.cpp (makeFindCommandForShards)
        • src/mongo/s/query/planner/cluster_aggregate.cpp (runAggregateImpl)
        • src/mongo/s/commands/query_cmd/cluster_distinct_cmd.cpp (prepareDistinctForPassthrough)
      • Router's own remaining budget - a decaying remainder from
        opCtx->getRemainingMaxTimeMillis():
        • src/mongo/db/pipeline/sharded_agg_helpers.cpp (~lines 1740-1743, in dispatchTargetedPipelineAndAddMergeCursors)

      Note the second overwrites the first on the aggregate targeted-dispatch path, so aggregate resolves a total in runAggregateImpl and then replaces it with a remainder before dispatch.

      Why this matters

      1. Inconsistent user-visible semantics

      Under the total convention the shard anchors a fresh full budget at its own start time; under the remainder convention it does not. So a query settings maxTimeMS currently behaves as per node on some paths and total across the operation on others.

      2. Inconsistent observability

      maxTimeMS as reported in shard-side profiler entries, slow-query logs and currentOp is a stable total on some paths and a decaying remainder on others, so it is not reliably comparable to what the client sent.

      Scope of Work

      First decide the intended semantics of a query settings maxTimeMS - per node or per operation.
      This is a user-visible decision, not purely a refactor.

      • If total is chosen: change sharded_agg_helpers.cpp to stop overwriting maxTimeMS with the remaining budget.
      • If remainder is chosen: change the three builders listed above, and remove the shard-side re-application in QuerySettingsShardService::lookupQuerySettingsWithRejectionCheck which re-grants a fresh budget.

      Then tighten the tripwire in async_requests_sender.cpp (cmdMaxTimeMS == qsMaxTimeMS || cmdMaxTimeMS >= remaining) to exact equality.

      Technical Notes

      The current "per node" semantics is encoded in an existing test assertion with very little margin. jstests/core/query/query_settings/query_settings_max_time_ms.js asserts assert.gte(Math.max(...durations), kLooseMaxTimeMS). Measured over 3 runs each in sharded_collections_jscore_passthrough_base, shard-side durationMillis against the then-2000ms threshold (the constants have since been raised, but the mechanism and margins are unchanged):

      • with the shard-side re-application (today): 2008-2011 (min margin +8ms)
      • without it: 2000-2011, one sample at exactly 2000 (min margin 0ms)

      So moving to the remainder convention requires deliberately updating that assertion, not just the production code - otherwise it becomes flaky rather than failing outright.

      Acceptance Criteria

      • All shard-bound query commands (find, aggregate, distinct) forward maxTimeMS using one documented convention.
      • The async_requests_sender.cpp tripwire is tightened to exact equality.
      • The intended per-node vs per-operation semantics is documented alongside applyMaxTimeMSFromSettings.
      • query_settings_max_time_ms.js and jstests/noPassthrough/query/query_settings_max_time_ms_readconcern_wait_race.js pass in both replica-set and sharded passthroughs.

            Assignee:
            Catalin Sumanaru
            Reporter:
            Daniel Tabacaru
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: