Issue + Relevant Info

Reported via MongoDB Support case. After a transient network disruption to the cluster, affected MongoClient instances never recover even once the cluster is fully reachable again: every replica-set member stays type=UNKNOWN, state=CONNECTING indefinitely, and only a JVM restart restores connectivity.

Root cause (confirmed with the Java driver team on Slack): the server-monitor handshake write leg has no effective timeout. This violates the CSOT spec, which requires that when timeoutMS is unset, socketTimeoutMS MUST bound socket reads and writes. The driver only enforces this for reads.

Impact

• Customer hit 101 of 141 application instances wedged for hours (all same deployment generation); instances started after the disruption connected normally. Recovery required a JVM restart. No ERROR log, no exception surfaced to the app, and no metric indicated the stuck state.
• Affects all async transports. Customer is on mongodb-driver-reactivestreams / kotlin 5.10.0.

Confirmed

• Thread dump (SIGQUIT, 83 min after the incident) shows monitor threads permanently parked (~80ms CPU over 83 min ⇒ parked, not spinning; no JVM deadlock) on an untimed CountDownLatch.await():
  • DefaultServerMonitor$ServerMonitor.run → setupNewConnectionAndGetInitialDescription → InternalStreamConnection.open → startHandshake → sendAndReceive → sendMessage → AsynchronousChannelStream.write → FutureAsyncCompletionHandler.getWrite() → get() → untimed latch.await().
• The latch is only released by completed() / failed(). AsynchronousSocketChannel.write is invoked with the write timeout, but that value is 0 for monitor connections, and a non-positive timeout is treated as "never time out" ⇒ the completion handler never fires ⇒ the latch is never released ⇒ the monitor thread parks forever ⇒ the server description is never updated back to a reachable type ⇒ topology stays UNKNOWN/CONNECTING until JVM restart.
• Code path for the zero value: DefaultServerMonitor.setupNewConnectionAndGetInitialDescription builds its OperationContext via operationContextFactory.create() → TimeoutSettings.connectionOnly() (passes null for timeoutMS). Then in TimeoutContext:

  public long getReadTimeoutMS()  { return timeoutOrAlternative(timeoutSettings.getReadTimeoutMS()); }
  public long getWriteTimeoutMS() { return timeoutOrAlternative(0); }
  

  Reads fall back to SocketSettings.readTimeoutMS; writes fall back to a hard-coded 0.

• Asymmetry confirms the read/write split: in the same JVM a second MongoClient to a different cluster through the same PrivateLink hostname (different ports) stayed healthy — its monitor threads were parked on getRead(), which does time out. A freshly started process reached the exact same ports of the affected cluster successfully at the same moment the stuck process was failing on them.
• CSOT-spec basis (per driver team): when timeoutMS is unset, socketTimeoutMS MUST bound reads and writes; the driver only bounds reads ⇒ nothing bounds the handshake write.
• No config mitigates it (confirmed by driver team): connectTimeoutMS only bounds connect; socketTimeoutMS only reaches the read path. External workarounds only (TCP keepalives, restart).

Hypothesis

• The stall is a frozen-flow / zero-window TCP socket: connect succeeds, but the flow freezes before the first handshake bytes are ACKed, so the write parks in await(). This is an unusual socket shape and is a plausible product of the PrivateLink disruption — it explains why the defect is rarely seen (small writes normally land in the kernel send buffer instantly; ordinary failures hit the bounded connect/read legs). Not yet directly proven; an ss -tino capture on an affected pod showing the wedged socket with a stalled send-queue / no keepalive progress would confirm the mechanism.

Investigation so far

• Reviewed with the Java driver team on Slack — agreed this is a defect, not by-design, and a CSOT-spec violation. It is long-standing: the driver never had write timeouts (blocking Java sockets can't enforce one), and when CSOT added enforcement to the async transports (5.2), the no-timeoutMS fallback kept the legacy "no write timeout" behavior.
• Searched for existing coverage: JAVA-1265, JAVA-3301 and JAVA-6264 are related but do not cover the monitor write-timeout gap. This is a new report.
• Prior support/internal context: HELP-95007 (Java driver monitor-thread behaviour post-8.0 upgrade), HELP-93816 (monitor/connection recovery), PROACTIVE-144779 / HELP-98966 (the cluster instability that acted as the trigger — repeated elections/restarts behind PrivateLink after an M60→M40 tier reduction under load).

Fix Scope / Caveat

The likely fix (write-timeout fallback to connectTimeoutMS) only helps the async transports: an async write is non-blocking, so the timeout is enforced in user space (a scheduled task fails the operation and closes the channel). A blocking write parks the thread inside the OS write() syscall, and the JVM does not expose a send timeout (SO_SNDTIMEO). Therefore the sync (SocketStream) and Unix-domain-socket transports remain exposed to this frozen-flow scenario even after the fix. The reporting customer is on the async reactive-streams driver, so a fix would cover their case.

Requested Action

• Confirm and triage this as a driver bug: the server-monitor handshake write is not bounded by any timeout when timeoutMS is unset (TimeoutContext.getWriteTimeoutMS() falls back to 0), causing async-transport monitor threads to park indefinitely and the topology to never recover.
• Assess making the write-timeout fall back to a non-zero value (e.g. connectTimeoutMS) for monitor connections so the handshake write is bounded, per the CSOT requirement that socketTimeoutMS bound both reads and writes.
• Advise on the sync / Unix-domain-socket transports, which remain exposed post-fix, and on any recommended interim guidance we can pass to the customer (currently: TCP keepalives / restart only).

Relevant Logs

// Application, during the stuck state:
Waiting for server to become available for operation ping with ID 9254.
Remaining time: 29999 ms. ... topology description: {type=REPLICA_SET, servers=[
  {address=…:1029, type=UNKNOWN, state=CONNECTING,
   exception={com.mongodb.MongoSocketReadTimeoutException: Timeout while receiving message},
   caused by {java.nio.channels.InterruptedByTimeoutException}}, …

// Thread dump (SIGQUIT, 83 min after incident) — 6 monitor threads permanently parked:
"cluster-<id>-<host>:1029" #61 daemon prio=5 cpu=79.94ms elapsed=4984.80s
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@25.0.3/Native Method)
	- parking to wait for <0x00000000daef8e38> (a java.util.concurrent.CountDownLatch$Sync)
	at java.util.concurrent.CountDownLatch.await(java.base@25.0.3/CountDownLatch.java:230)
	at com.mongodb.internal.connection.AsynchronousChannelStream$FutureAsyncCompletionHandler.get(AsynchronousChannelStream.java:314)
	at com.mongodb.internal.connection.AsynchronousChannelStream$FutureAsyncCompletionHandler.getWrite(AsynchronousChannelStream.java:305)
	at com.mongodb.internal.connection.AsynchronousChannelStream.write(AsynchronousChannelStream.java:131)
	at com.mongodb.internal.connection.InternalStreamConnection.sendMessage(InternalStreamConnection.java:887)
	…
	at com.mongodb.internal.connection.InternalStreamConnectionInitializer.startHandshake(InternalStreamConnectionInitializer.java:78)
	at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:236)
	at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.setupNewConnectionAndGetInitialDescription(DefaultServerMonitor.java:282)
	at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.run(DefaultServerMonitor.java:203)

Environment: mongo-java-driver / kotlin / reactive-streams 5.10.0 (source also inspected in 5.9.2); JDK Amazon 25.0.3+9-LTS; Kotlin 2.4.10; Linux 6.18.41-94.142.amzn2023.aarch64 (aarch64), Kubernetes; all client timeouts at defaults.