-
Type:
Bug
-
Resolution: Duplicate
-
Priority:
Major - P3
-
None
-
Affects Version/s: 8.2.12, 8.3.13, 9.0.4
-
Component/s: None
-
None
-
Networking & Observability
-
ALL
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
- Description
We observed continuous, approximately linear memory growth on MongoDB 8.3.11 nodes running long-lived async exhaust streams, particularly Secondary and Hidden replica set members.
The process memory grows by approximately 0.5–0.6 MiB per hour and does not stabilize while the exhaust stream remains active. Comparable MongoDB 7.0 deployments did not show the same behavior.

-
- Heap Profile Evidence
The growing allocations are associated with the following path:
SharedStateImpl::addChild
-> SharedSemiFuture::toFutureImpl
-> CancellationToken::onCancel
-> CancellationSource::CancellationSource
-> AsyncDBClient::_withCancellation
-> AsyncDBClient::_waitForResponse
-> AsyncDBClient::awaitExhaustCommand
-> ExhaustResponseReaderTL::_read
-
- Suspected Root Cause
`ExhaustResponseReaderTL::_read()` passes the same stream-level cancellation token to every exhaust read:
awaitExhaustCommand(_baton, _cancelSource.token());
For every response, `AsyncDBClient::_withCancellation()` creates a new `OperationState` containing a child `CancellationSource` derived from this token.
Creating that child source subscribes to the parent token's `onCancel()` future and adds a child shared state through `SharedStateImpl::addChild()`.
Because streamable hello may remain active indefinitely, the parent cancellation state does not complete. Its child Future nodes therefore accumulate for the lifetime of the stream, producing unbounded memory growth proportional to the number of responses.
There is also a related issue in `ExhaustResponseReaderTL::next()`: a per-read timeout token is created, but `_read()` does not receive it and continues using the stream-level token.
-
- Expected Behavior
Memory used by cancellation state should remain bounded regardless of how many responses are received on a long-lived exhaust stream.
-
- Suggested Fix
Use a short-lived, independent `CancellationSource` for each read and pass its token to `_read()` / `awaitExhaustCommand()`.
The stream-level cancellation token should register only one callback, which forwards cancellation to the currently active per-read source. The per-read cancellation state can then be released after each response.
This also allows the existing local timeout to cancel the actual in-flight read.
If this analysis is confirmed, I would be happy to prepare and submit a patch with regression tests.
- duplicates
-
SERVER-135020 Memory leak: per-response CancellationSource child states accumulate unboundedly on long-lived parent tokens in exhaust command readers (ExhaustResponseReaderTL / AsyncDBClient::_withCancellation)
-
- Open
-
- is related to
-
SERVER-136787 Streamable ReplicaSetMonitor accumulates hundreds-thousands of concurrent awaitable hello streams per target after NetworkInterfaceExceededTimeLimit, leading to OOM on arbiters / data nodes
-
- Needs Verification
-
- related to
-
SERVER-135020 Memory leak: per-response CancellationSource child states accumulate unboundedly on long-lived parent tokens in exhaust command readers (ExhaustResponseReaderTL / AsyncDBClient::_withCancellation)
-
- Open
-
-
SERVER-103841 Memory leak in TransactionCoordinator associated to long-lived cancellation source
-
- Closed
-
-
SERVER-92333 Audit use of long lived CancellationSources
-
- Blocked
-
-
SERVER-113039 Stop attaching to the root cancelToken in resharding
-
- Closed
-
-
SERVER-118807 Create hierarchy class for cancellable tokens for resharding
-
- Closed
-
-
SERVER-103945 Better memory management for CancellationToken
-
- Backlog
-