ExportXMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Execution
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Query memory load shedding as designed may be at risk of shedding a shard-side or secondary $merge/$out after it has written.

      Problem

      Query memory load shedding (SERVER-130333) protects writes with a "write-intent latch": an operation is never shed once its Locker has taken a global IX/X lock (wasGlobalLockTakenForWrite()). That protects local writes, but not writes an operation sends to another node, which never take a global IX on the shed operation's own Locker. As a result, a $merge/$out running on a shard (or a secondary) can be shed with the retryable QueryMemoryLimitExceeded after some batches have already been written.

      This is unverified (found by code inspection) and needs investigation and a repro.

      Suspected path (shard)

      1. A user sends aggregate [..., \{$group: ...\}, \{$merge: ...\}] to mongos. The router correctly leaves write pipelines ineligible (cluster_aggregate_cmd.h, if (!hasWriteStage) markOperationQueryMemorySheddingEligible(opCtx)).
      2. Mongos sends the pipeline, including $merge, to the shard(s) as an aggregate command. On the shard, aggregate_command.cpp:228 calls markOperationQueryMemorySheddingEligible(opCtx) unconditionally, even for pipelines with a write stage.
      3. The request comes from an internal client, so the shard uses ShardServerProcessInterface (mongod_process_interface_factory.cpp). Its insert/update (shardsvr_process_interface.cpp:188-247) call cluster::write(expCtx->getOperationContext(), ...), which targets the owning shards and sends separate insert/update commands over the network (BatchWriteExec / unified write executor). No global lock is taken on this opCtx along that path. The global IX is taken by the receiving command's own opCtx, even when the target is the same shard.
      4. The aggregation's Locker therefore only ever holds IS, the latch is never set, and the op stays eligible. Under memory pressure it can be shed at a later read checkpoint (queryMemoryCheckLoadShedding via ExpressionContext/SBE/PlanYieldPolicy) after earlier $merge batches have been written.

      The same likely applies to:

      • $merge/$out on a replica-set secondary: ReplicaSetNodeProcessInterface forwards writes to the primary via _executeCommandOnPrimary.
      • $out setup/teardown on a shard: createTempCollection via writeToLocalShard, plus index creation and rename via the router.
      • analyzeShardKey split-point inserts when the node is not primary (QueryAnalysisClient::insert remote path). Low impact: these are scratch docs with a TTL.

      Impact

      A client receives a retryable error after a partial, non-idempotent $merge/$out, and a naive retry can re-apply writes. The feature is off by default (queryMemoryLoadSheddingLowMarkPercent: -1), and sharded and secondary write aggregations are not yet a supported target, so this is not urgent. But it must be fixed before load shedding is enabled for sharded clusters.

      Investigation / repro

      • Sharded jstest: run $merge into a sharded or unsharded target with queryMemoryLoadSheddingAlwaysShed enabled after the first batch, e.g. a failpoint between WriterStage flushes. Assert whether the op fails with QueryMemoryLimitExceeded while the target already contains documents.
      • Repeat against a secondary with readPreference: secondary.
      • Confirm there is no in-process shortcut for same-shard targets in cluster::write / the AsyncRequestsSender.

      Possible fix

      • Simplest: don't mark the shard-side aggregate/getMore eligible when the pipeline has a write stage (mirror cluster_aggregate's aggHasWriteStage check). Downside: the read phase of every $merge/$out becomes unsheddable, and getMore would need the write-stage information stashed on the cursor.
      • More precise: add a one-way "write started" flag to LoadSheddingOpState, set in DocumentSourceWriteBlock's constructor (every $merge/$out write, including $out initialize/finalize, runs inside one). Follow it with opCtx->checkForInterrupt(), and have isShedCandidate() reject ops with the flag set. With seq_cst ordering, either the shedder sees the flag, or the op sees the kill and fails before writing. This keeps the read phase sheddable and covers local, remote, in-thread and queued-op sweep sheds alike.

            Assignee:
            Unassigned
            Reporter:
            Steve Tarzia
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: