Summary

Applications using the reactive driver with the Netty transport slowly lose memory under load: heap and direct memory grow with traffic, full GCs reclaim nothing, and the process eventually dies with OutOfMemoryError ("Cannot reserve ... direct buffer memory" in our case). Netty's leak detector stays silent, because these are genuine strong references, not missed release() calls. And applications are exposed without ever opting in: Spring Boot's MongoReactiveAutoConfiguration switches the driver to the Netty transport whenever Netty is present in the classpath, which appears to always be the case for a WebFlux application.

In our production case (an Appsmith server: Spring WebFlux, multi-megabyte MongoDB documents), 15 minutes of ordinary traffic under 150 ms network latency left the object graphs of ~50 completed HTTP requests unreclaimable (~580 MB of direct memory plus ~1.6 GB of retained heap, surviving every full GC and 10 minutes of idle), ending in recurring production OutOfMemoryError.

The cause: when the Netty transport opens a pooled connection, the operation that was waiting for that connection stays anchored in memory for as long as the connection lives. The operation finishes normally and its caller gets its result, but its whole object graph (pool acquisition callbacks, reactive-streams subscribers, and up to 5.5.x even the decoded result documents of the last cursor batch) can never be garbage-collected.

How it happens

1. When a channel finishes opening, OpenChannelFutureListener.operationComplete() installs a listener on channel.closeFuture(). Its job: when the connection eventually dies, fail the read waiting on it, and any later read attempt, with "The connection to the server was closed" (it calls handleReadResponse(null, ioException), which sets pendingException and completes the pendingReader). That listener legitimately lives as long as the channel.
2. To do its job, the listener's lambda only needs the enclosing NettyStream (it calls handleReadResponse(...)). But the lambda is written inside OpenChannelFutureListener, a non-static inner class of NettyStream, so javac resolves the call as NettyStream.this.handleReadResponse(...) and makes the lambda capture the whole listener instance, just to reach the stream through it (verified in heap dumps: the synthetic lambda's arg$1 field references the OpenChannelFutureListener).
3. That instance has a handler field: the AsyncCompletionHandler that was waiting for the connection to open. It fires once, when the open completes, and is never needed again. But since the close listener (lifetime: the channel) now references it, the handler and everything reachable from it survives every GC until the channel closes, which in a pool is typically never.

The offending line, in com.mongodb.internal.connection.netty.NettyStream (present from at least 4.x up to current master):

// OpenChannelFutureListener.operationComplete()
channel = channelFuture.channel();
channel.closeFuture().addListener((ChannelFutureListener) future1 ->
        handleReadResponse(null, new IOException("The connection to the server was closed")));
...
handler.completed(null);

Please provide the version of the driver. If applicable, please provide the MongoDB server version and topology (standalone, replica set, or sharded cluster).

Driver: reproduced on 5.0.1, 5.5.2 and 5.9.0 (latest release), mongodb-driver-reactivestreams; the offending capture lives in driver-core's Netty transport and is unchanged up to current master (5.10.0-SNAPSHOT).

MongoDB server: 7.0.x, single-node replica set (also reproduces against a standalone; the defect is purely client-side and needs no particular topology).

How to Reproduce

Standalone reproducer attached (NettyPinningRepro.java + pom.xml, flat files, runs in ~2 minutes):

mvn compile exec:java -Dexec.args="netty" -Ddriver.version=5.9.0

TWO WAVES of 64 concurrent find().collectList() reads of 1 MB documents against the same client: wave 1 starts on a cold pool and warms the pool and every transport cache; wave 2 runs on the warm pool and measures what a steady application keeps accumulating. All harness-side strong references are dropped after each wave and full GCs are forced. Two WeakReference canary families per operation test reachability (the JVM clears weak references during marking as soon as no strong path exists): PIPELINE STATE, a 1 MB ballast captured by a lambda of the operation's own reactive pipeline (standing for whatever a real application keeps reachable from its subscriber chain: request context, buffers), and the LAST-BATCH DOCUMENTS kept by the cursor.

On the latest release, 5.9.0:

$ mvn compile exec:java -Dexec.args="netty" -Ddriver.version=5.9.0
transport=netty | 2 waves of 64 concurrent full-collection reads (40 x 1024 KB docs)
wave 1 completed (every response was delivered)
after wave 1 (cold pool) + GC | used heap 44 MB
    wave 1: pipeline state (1 MB/op) pinned 26/64, last-batch documents pinned 0/64
wave 2 completed (every response was delivered)
after wave 2 (warm pool) + GC | used heap 52 MB
    wave 1: pipeline state (1 MB/op) pinned 26/64, last-batch documents pinned 0/64
    wave 2: pipeline state (1 MB/op) pinned 7/64, last-batch documents pinned 0/64
waiting 95 s, client still open...
after idle, still open        | used heap 52 MB
    wave 1: pipeline state (1 MB/op) pinned 26/64, last-batch documents pinned 0/64
    wave 2: pipeline state (1 MB/op) pinned 7/64, last-batch documents pinned 0/64
after client.close() + GC     | used heap 18 MB
    wave 1: pipeline state (1 MB/op) pinned 0/64, last-batch documents pinned 0/64
    wave 2: pipeline state (1 MB/op) pinned 0/64, last-batch documents pinned 0/64

$ mvn compile exec:java -Dexec.args="default" -Ddriver.version=5.9.0
transport=default | 2 waves of 64 concurrent full-collection reads (40 x 1024 KB docs)
wave 1 completed (every response was delivered)
after wave 1 (cold pool) + GC | used heap 372 MB
    wave 1: pipeline state (1 MB/op) pinned 0/64, last-batch documents pinned 0/64
wave 2 completed (every response was delivered)
after wave 2 (warm pool) + GC | used heap 712 MB
    wave 1: pipeline state (1 MB/op) pinned 0/64, last-batch documents pinned 0/64
    wave 2: pipeline state (1 MB/op) pinned 0/64, last-batch documents pinned 0/64
waiting 95 s, client still open...
after idle, still open        | used heap 12 MB
    wave 1: pipeline state (1 MB/op) pinned 0/64, last-batch documents pinned 0/64
    wave 2: pipeline state (1 MB/op) pinned 0/64, last-batch documents pinned 0/64
after client.close() + GC     | used heap 12 MB
    wave 1: pipeline state (1 MB/op) pinned 0/64, last-batch documents pinned 0/64
    wave 2: pipeline state (1 MB/op) pinned 0/64, last-batch documents pinned 0/64

How to read this: with the Netty transport, the pipeline state of every operation that overlapped a connection opening stays strongly reachable after full GCs, and wave 2 ADDS its own pinned graphs on top of wave 1's (26, then 26+7 here; the counts follow pool-growth timing): that is the accumulation a live application experiences on every load episode, since the pool keeps growing towards maxPoolSize. The pinning is unchanged after 95 s of idle with the client open, and is released the instant the client (hence the channels) is closed: proof that the anchor is the channel close listener. With the default transport nothing is ever pinned across either wave; its elevated heap figure right after the waves is the driver's global static PowerOfTwoBufferPool.DEFAULT reserve (heap buffers), which prunes itself during the idle period, client still open (712 MB -> 12 MB in the same run): a healthy bounded cache, unlike the Netty-transport pinning.

On 5.0.1 (and the whole pre-5.6 range, i.e. what Spring Boot 3.x managed up to 5.5.x), the DOCUMENT canaries light up on top of the pipeline ones, because AsyncCommandBatchCursor.commandCursorResult used to keep the last decoded batch reachable from the same pinned graph:

$ mvn compile exec:java -Dexec.args="netty" -Ddriver.version=5.0.1
transport=netty | 2 waves of 64 concurrent full-collection reads (40 x 1024 KB docs)
wave 1 completed (every response was delivered)
after wave 1 (cold pool) + GC | used heap 327 MB
    wave 1: pipeline state (1 MB/op) pinned 28/64, last-batch documents pinned 28/64
wave 2 completed (every response was delivered)
after wave 2 (warm pool) + GC | used heap 472 MB
    wave 1: pipeline state (1 MB/op) pinned 28/64, last-batch documents pinned 28/64
    wave 2: pipeline state (1 MB/op) pinned 13/64, last-batch documents pinned 13/64
waiting 95 s, client still open...
after idle, still open        | used heap 472 MB
    wave 1: pipeline state (1 MB/op) pinned 28/64, last-batch documents pinned 28/64
    wave 2: pipeline state (1 MB/op) pinned 13/64, last-batch documents pinned 13/64
after client.close() + GC     | used heap 20 MB
    wave 1: pipeline state (1 MB/op) pinned 0/64, last-batch documents pinned 0/64
    wave 2: pipeline state (1 MB/op) pinned 0/64, last-batch documents pinned 0/64

With the suggested one-line fix applied to current main (5.10.0-SNAPSHOT), the same netty run reports 0/64 for both families, in both waves, at every step, and used heap stays at 17-19 MB throughout.

Additional Background

The retained chain goes remarkably far: handler -> DefaultConnectionPool$OpenConcurrencyLimiter lambdas -> operation callback -> MonoCreate$DefaultMonoSink -> the driver-internal BatchCursorFlux subscription -> the caller's downstream subscribers, and up to 5.5.x the decoded documents kept in AsyncCommandBatchCursor.commandCursorResult. Since 5.6.0 that last hop is gone: JAVA-5940, an improvement about open cursors needlessly retaining their already-delivered batch, incidentally unlinked the documents from this pinned graph too. It was not aimed at this defect (the ticket mentions neither the transport nor any anchoring mechanism) and left the anchor itself in place: this is why the document canary family reads 0 on recent versions while the graph itself is still pinned.

Real-workload impact (attached chart: oldgen-comparison-appsmith-bench.png)

The attached chart shows the same defect measured on a real application (an Appsmith CE server, code unchanged, only the driver build swapped): three identical 30-minute bench campaigns, 8 concurrent readers replaying the application's heaviest read endpoints against a production-sized dataset, MongoDB behind a 150 ms (+/- 100 ms) latency proxy and ordinary connection churn (maxIdleTimeMS=20000, so the pool keeps closing and reopening connections, which is what any long-running service experiences), each followed by 10 minutes of complete silence. The curve is the G1 Old Gen post-GC floor (2-minute rolling minimum of the used size), i.e. what full GCs cannot reclaim. After the silence, the floor settles at ~1.15 GB with 5.5.2, at ~460 MB with 5.9.0 (JAVA-5940 shrank what each pinned graph drags along, not the pinning itself), and returns to the ~90 MB baseline with the one-line fix.

Suggested fix

Avoid capturing the inner class in the close-future listener - capture only the enclosing stream (one-line change):

NettyStream stream = NettyStream.this;
channel.closeFuture().addListener((ChannelFutureListener) future1 ->
        stream.handleReadResponse(null, new IOException("The connection to the server was closed")));

Defensively, the handler field could also be cleared after handler.completed(null) / handler.failed(...), and/or the close listener removed when the stream is closed gracefully.

I will open a GitHub pull request implementing this fix (with a regression test) very shortly.