-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Query Execution
-
ALL
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Problem
BlockingResultsMerger::kill() calls _arm->kill(opCtx).wait(). The .wait() uses Interruptible::notInterruptible(), which does a plain cv.wait() and never polls the baton.
AsyncResultsMerger::kill() calls _cancellationSource.cancel() to cancel any pending retry delay timer. However, AsioNetworkingBaton::waitUntil registers timer cancellation via .thenRunOn(baton) (src/mongo/transport/asio/asio_networking_baton.cpp line 278). This means the _cancelTimer callback is queued as a task on the baton — it requires the baton to be polled to actually execute.
Since .wait() never polls the baton:
- The _cancelTimer callback sits in the baton's queue
- The waitUntil promise never resolves
- The retry callback never fires
- _cleanUpKilledBatch never runs
- The kill future is never signaled
- .wait() hangs forever
Reproduction
This occurs when:
- An ARM has a pending retry delay (scheduled via _subBaton->waitUntil(now + delay, token))
- BlockingResultsMerger::kill() is called (e.g. from ClusterClientCursorGuard::~ClusterClientCursorGuard())
- The test or operation hangs indefinitely in kill()
- Observed in sharded query tests using the failIngressRequestRateLimiting failpoint, where a retryable error triggers the ARM retry path followed by cursor cleanup.
Stack Trace
#0 __futex_abstimed_wait_cancelable64 #1 pthread_cond_wait #2 condition_variable_any::wait #3 NotInterruptible::waitForConditionOrInterruptNoAssertUntil #6 SharedStateBase::wait(Interruptible*) #7 BlockingResultsMerger::kill(OperationContext*) #8 ClusterClientCursorImpl::kill(OperationContext*) #9 ClusterClientCursorGuard::~ClusterClientCursorGuard() #10 cluster_aggregation_planner::runPipelineOnMongoS(...)
Fix
Replace .wait() with future.get(opCtx) wrapped in runWithoutInterruptionExceptAtGlobalShutdown. The opCtx-based wait polls the baton, allowing the cancellation callback to execute. runWithoutInterruptionExceptAtGlobalShutdown is needed because kill() is called from destructors where the opCtx may already be interrupted — a plain .get(opCtx) would throw before polling the baton, and throwing from a destructor terminates the process.
// BEFORE: void BlockingResultsMerger::kill(OperationContext* opCtx) { _arm->kill(opCtx).wait(); } // AFTER: void BlockingResultsMerger::kill(OperationContext* opCtx) { auto future = _arm->kill(opCtx); opCtx->runWithoutInterruptionExceptAtGlobalShutdown([&] { future.get(opCtx); }); }
- blocks
-
SERVER-114130 Move the failRateLimiting fail point to the ingress request rate limiter in session workflow
-
- In Progress
-