BlockingResultsMerger::kill() hangs when ARM has pending retry delay because baton is never polled

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Execution
    • ALL
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Problem

      BlockingResultsMerger::kill() calls _arm->kill(opCtx).wait(). The .wait() uses Interruptible::notInterruptible(), which does a plain cv.wait() and never polls the baton.

      AsyncResultsMerger::kill() calls _cancellationSource.cancel() to cancel any pending retry delay timer. However, AsioNetworkingBaton::waitUntil registers timer cancellation via .thenRunOn(baton) (src/mongo/transport/asio/asio_networking_baton.cpp line 278). This means the _cancelTimer callback is queued as a task on the baton — it requires the baton to be polled to actually execute.

      Since .wait() never polls the baton:

      1. The _cancelTimer callback sits in the baton's queue
      2. The waitUntil promise never resolves
      3. The retry callback never fires
      4. _cleanUpKilledBatch never runs
      5. The kill future is never signaled
      6. .wait() hangs forever

      Reproduction

      This occurs when:

      • An ARM has a pending retry delay (scheduled via _subBaton->waitUntil(now + delay, token))
      • BlockingResultsMerger::kill() is called (e.g. from ClusterClientCursorGuard::~ClusterClientCursorGuard())
      • The test or operation hangs indefinitely in kill()
      • Observed in sharded query tests using the failIngressRequestRateLimiting failpoint, where a retryable error triggers the ARM retry path followed by cursor cleanup.

      Stack Trace

      #0  __futex_abstimed_wait_cancelable64
      #1  pthread_cond_wait
      #2  condition_variable_any::wait
      #3  NotInterruptible::waitForConditionOrInterruptNoAssertUntil
      #6  SharedStateBase::wait(Interruptible*)
      #7  BlockingResultsMerger::kill(OperationContext*)
      #8  ClusterClientCursorImpl::kill(OperationContext*)
      #9  ClusterClientCursorGuard::~ClusterClientCursorGuard()
      #10 cluster_aggregation_planner::runPipelineOnMongoS(...)
      

      Fix

      Replace .wait() with future.get(opCtx) wrapped in runWithoutInterruptionExceptAtGlobalShutdown. The opCtx-based wait polls the baton, allowing the cancellation callback to execute. runWithoutInterruptionExceptAtGlobalShutdown is needed because kill() is called from destructors where the opCtx may already be interrupted — a plain .get(opCtx) would throw before polling the baton, and throwing from a destructor terminates the process.

      // BEFORE:
      void BlockingResultsMerger::kill(OperationContext* opCtx) {
      _arm->kill(opCtx).wait();
      }
      
      // AFTER:
      void BlockingResultsMerger::kill(OperationContext* opCtx) {
      auto future = _arm->kill(opCtx);
      opCtx->runWithoutInterruptionExceptAtGlobalShutdown([&] { future.get(opCtx); });
      }
      

            Assignee:
            Unassigned
            Reporter:
            Blake Oler
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: