-
Type:
Bug
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: 4.19.0
-
Component/s: None
-
None
-
None
-
Python Drivers
-
Not Needed
-
-
None
-
None
-
None
-
None
-
None
-
None
Description
We hit this in production when a replica set node ran out of memory (OOM) and restarted. The client kept failing server selection and only recovered after the client process was restarted. We verified that the bug described below was the cause.
If a server's own hello response removes it from the topology, _update_servers() closes that server from inside its own monitor task. AsyncPeriodicExecutor.close() then cancels the current task:
AsyncPeriodicExecutor._run -> Monitor._run -> Topology.on_change -> _process_change -> _update_servers -> Server.close -> Monitor.close -> AsyncPeriodicExecutor.close -> self._task.cancel()
CancelledError is raised at the next await (in Pool._reset), which has two effects:
- self._servers.pop(address) never runs, so the old Server stays in Topology._servers.
- The monitor task ends cancelled, and AsyncPeriodicExecutor.open() will not restart it.
When the primary's hello lists the address again, _update_servers() reuses the old Server. Its monitor never runs again, so the server stays Unknown with no error. Once every member has gone through this, the client cannot recover without a restart.
Self-removal reproduced with a standalone response, a mongos response, a me mismatch and a setName mismatch. The synchronous MongoClient recovered in the same standalone scenario.
Steps to reproduce
Run with pip install pymongo==4.19.0. No MongoDB server is needed.
import asyncio, struct import bson from pymongo import AsyncMongoClient HOSTS = ["127.0.0.1:27101", "127.0.0.1:27102"] role = {27101: "secondary", 27102: "primary"} def hello(port): doc = {"ok": 1, "minWireVersion": 0, "maxWireVersion": 21} if role[port] == "standalone": return doc return doc | {"setName": "rs", "hosts": HOSTS, "me": f"127.0.0.1:{port}", "primary": HOSTS[1], "secondary": role[port] == "secondary", "isWritablePrimary": role[port] == "primary"} def server(port): async def handle(reader, writer): try: while True: length, req_id, _, _ = struct.unpack("<iiii", await reader.readexactly(16)) body = await reader.readexactly(length - 16) cmd = next(iter(bson.decode(body[5:]))).lower() reply = hello(port) if cmd in ("hello", "ismaster") else {"ok": 1} data = b"\0" * 5 + bson.encode(reply) writer.write(struct.pack("<iiii", 16 + len(data), 0, req_id, 2013) + data) except (asyncio.IncompleteReadError, ConnectionError): writer.close() return asyncio.start_server(handle, "127.0.0.1", port) async def main(): servers = [await server(p) for p in role] client = AsyncMongoClient(",".join(HOSTS), replicaSet="rs", heartbeatFrequencyMS=500, serverMonitoringMode="poll", serverSelectionTimeoutMS=2000) await client.admin.command("ping") for port, original in role.items(): # each node briefly removes itself, then recovers role[port] = "standalone" await asyncio.sleep(2) role[port] = original await asyncio.sleep(2) try: await client.admin.command("ping") print("recovered") except Exception as exc: print(f"{type(exc).__name__}: {exc}") asyncio.run(main())
Expected: prints recovered.
Actual:
ServerSelectionTimeoutError: No replica set members found yet, Timeout: 2.0s, Topology Description: <TopologyDescription id: ..., topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('127.0.0.1', 27101) server_type: Unknown, rtt: None>]>
Suggested fix
Do not cancel the current task in AsyncPeriodicExecutor.close(). _stopped still ends the loop:
if self._task is not None and self._task is not asyncio.current_task(): self._task.cancel()
With this change, applied as a monkeypatch, the script prints recovered on 4.19.0 and main.
Popping the server before await server.close() in _update_servers() also recovered on 4.19.0, but the task still cancels itself, so it only works as extra hardening.