-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Unknown
-
None
-
Affects Version/s: 6.20.0, 6.22.0, 7.9.1
-
Component/s: Connection Layer
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Summary
If a connection is cleaned up (socketTimeoutMS fires, or the pool interrupts in-use connections after a heartbeat timeout) while Connection.writeCommand waits for the socket's drain event, the operation's promise never settles. No commandFailed event is emitted, and the connection is never checked back in, so the pool permanently loses one connection.
Cause
- writeCommand (src/cmap/connection.ts) waits on once(this.socket, 'drain', options) when socket.write() returns false. That is any command at least as large as the socket's writable high-water mark (16 KiB) that hasn't been flushed yet.
- utils.once settles only on drain or error.
- Connection.cleanup(error) calls this.socket.destroy() without the error. A destroyed socket emits close, not error or drain.
- this.dataEvents is still null during the write, so dataEvents?.throw(error) rejects nothing.
- The same happens for every path into cleanup during a pending drain: onTimeout (socketTimeoutMS), onError from interruptInUseConnections (PoolClearedOnNetworkError), onClose, and destroy.
- With CSOT enabled, timeoutForSocketWrite races the drain wait, so timeoutMS users are not affected.
Impact
Without socketTimeoutMS, a dead peer eventually surfaces as a TCP error on the socket, which rejects the drain wait. With socketTimeoutMS set, the timeout destroys the socket first, so the operation waits forever instead of failing.
We hit this in production on an Atlas sharded cluster. During a ~20 s network outage, heartbeats to several mongos timed out and socketTimeoutMS (10 s) fired on in-flight operations. A few application requests never completed and held their pool connections until the processes restarted 12+ hours later.
Repro
The script below starts a fake server on 127.0.0.1. The server answers the handshake, then stops reading in the middle of an 8 MiB insert, like a peer that disappeared without closing the TCP connection. The client uses socketTimeoutMS: 1000 and maxPoolSize: 1.
npm install mongodb && node repro.js
- Expected: insertOne rejects with MongoNetworkTimeoutError after about 1 s.
- Actual: insertOne is still pending when a 10 s watchdog fires. No commandFailed event is emitted. pool.currentCheckedOutCount stays at 1, and the next operation on that pool fails with MongoWaitQueueTimeoutError.
Same result on 6.20.0, 6.22.0, 7.9.1 and nightly 7.9.1-dev.20261120.sha.7a75a3c1 (Node 24.22.1).
// An operation never settles when socketTimeoutMS fires while writeCommand waits for 'drain'. // A fake server completes the handshake, then stops reading in the middle of a large insert, like a peer that went // away without closing the TCP connection. // Run: npm install mongodb && node repro.js (exit 1 = hung, exit 0 = rejected as expected) const net = require('node:net'); const { MongoClient, BSON } = require('mongodb'); const OP_REPLY = 1; const OP_QUERY = 2004; const OP_MSG = 2013; const SOCKET_TIMEOUT_MS = 1000; const helloReply = () => ({ ok: 1, isWritablePrimary: true, ismaster: true, maxBsonObjectSize: 16 * 1024 * 1024, maxMessageSizeBytes: 48 * 1000 * 1000, maxWriteBatchSize: 100000, localTime: new Date(), minWireVersion: 0, maxWireVersion: 21, }); const encode = (opCode, responseTo, doc) => { const body = BSON.serialize(doc); const prefix = opCode === OP_MSG ? Buffer.alloc(16 + 5) // header, flagBits, section kind 0 : Buffer.alloc(16 + 20); // header, responseFlags, cursorId, startingFrom, numberReturned prefix.writeInt32LE(prefix.length + body.length, 0); prefix.writeInt32LE(responseTo, 8); prefix.writeInt32LE(opCode, 12); if (opCode === OP_REPLY) { prefix.writeInt32LE(1, 32); } return Buffer.concat([prefix, body]); }; const server = net.createServer((socket) => { socket.on('error', () => {}); let pending = Buffer.alloc(0); const onData = (chunk) => { pending = Buffer.concat([pending, chunk]); const length = pending.length >= 4 ? pending.readInt32LE(0) : Infinity; if (length > 64 * 1024 && pending.length < length) { // stop reading mid-message: the client's socket buffers fill and its write waits for 'drain' socket.off('data', onData); socket.pause(); return; } if (pending.length < length) { return; } const requestId = pending.readInt32LE(4); const opCode = pending.readInt32LE(12); pending = pending.subarray(length); // every small message during the handshake is hello/isMaster/ping socket.write(encode(opCode === OP_QUERY ? OP_REPLY : OP_MSG, requestId, helloReply())); }; socket.on('data', onData); }); server.listen(0, '127.0.0.1', async () => { const client = new MongoClient(`mongodb://127.0.0.1:${server.address().port}/?directConnection=true`, { socketTimeoutMS: SOCKET_TIMEOUT_MS, maxPoolSize: 1, retryWrites: false, }); await client.connect(); const insert = client .db('test') .collection('test') .insertOne({ blob: 'x'.repeat(8 * 1024 * 1024) }) .then( () => 'resolved', (error) => `rejected with ${error.name}`, ); const watchdog = new Promise((resolve) => setTimeout(() => resolve('HUNG'), 10 * SOCKET_TIMEOUT_MS)); const outcome = await Promise.race([insert, watchdog]); console.log(`insertOne after ${10 * SOCKET_TIMEOUT_MS} ms watchdog: ${outcome}`); process.exit(outcome === 'HUNG' ? 1 : 0); });
Possible fix
Destroy the socket with the error, so a pending once(socket, 'drain') rejects:
private cleanup(error: Error): void { if (this.closed) { return; } this.socket.destroy(error); ... }
With this change, the repro rejects with MongoNetworkTimeoutError, which is retryable, on 6.20.0 and 7.9.1. Another option is to have the drain wait also settle when the connection emits close.
Related
NODE-6370 has the same symptom (an operation never resolves and its connection stays out of the pool) and was fixed in 6.10.0. This repro still hangs on 6.20.0 and later, so this is a separate path.
- related to
-
NODE-6370 bulkWrite does not resolve after update, and connection not returned to pool
-
- Closed
-