ExportXMLWordPrintableJSON

    • Type: Bug
    • Resolution: Duplicate
    • Priority: Major - P3
    • None
    • Affects Version/s: 8.2.12, 8.3.13, 9.0.4
    • Component/s: None
    • None
    • Networking & Observability
    • ALL
    • None
    • None
    • None
    • None
    • None
    • None
    • None

        1. Description

      We observed continuous, approximately linear memory growth on MongoDB 8.3.11 nodes running long-lived async exhaust streams, particularly Secondary and Hidden replica set members.

      The process memory grows by approximately 0.5–0.6 MiB per hour and does not stabilize while the exhaust stream remains active. Comparable MongoDB 7.0 deployments did not show the same behavior.

        1. Heap Profile Evidence

      The growing allocations are associated with the following path:

      SharedStateImpl::addChild
        -> SharedSemiFuture::toFutureImpl
        -> CancellationToken::onCancel
        -> CancellationSource::CancellationSource
        -> AsyncDBClient::_withCancellation
        -> AsyncDBClient::_waitForResponse
        -> AsyncDBClient::awaitExhaustCommand
        -> ExhaustResponseReaderTL::_read

        1. Suspected Root Cause

      `ExhaustResponseReaderTL::_read()` passes the same stream-level cancellation token to every exhaust read:

          awaitExhaustCommand(_baton, _cancelSource.token());

      For every response, `AsyncDBClient::_withCancellation()` creates a new `OperationState` containing a child `CancellationSource` derived from this token.

      Creating that child source subscribes to the parent token's `onCancel()` future and adds a child shared state through `SharedStateImpl::addChild()`.

      Because streamable hello may remain active indefinitely, the parent cancellation state does not complete. Its child Future nodes therefore accumulate for the lifetime of the stream, producing unbounded memory growth proportional to the number of responses.

      There is also a related issue in `ExhaustResponseReaderTL::next()`: a per-read timeout token is created, but `_read()` does not receive it and continues using the stream-level token.

        1. Expected Behavior

      Memory used by cancellation state should remain bounded regardless of how many responses are received on a long-lived exhaust stream.

        1. Suggested Fix

      Use a short-lived, independent `CancellationSource` for each read and pass its token to `_read()` / `awaitExhaustCommand()`.

      The stream-level cancellation token should register only one callback, which forwards cancellation to the currently active per-read source. The per-read cancellation state can then be released after each response.

      This also allows the existing local timeout to cancel the actual in-flight read.

      If this analysis is confirmed, I would be happy to prepare and submit a patch with regression tests.

        1. image-2026-10-15-21-21-03-171.png
          image-2026-10-15-21-21-03-171.png
          193 kB
        2. repro.sh
          4 kB

            Assignee:
            Michael Nugent
            Reporter:
            Jinzhong Li (EXT)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: