ExportXMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fixed
    • Priority: Major - P3
    • 4.20.0
    • Affects Version/s: 4.19.0
    • Component/s: None
    • None
    • None
    • Python Drivers
    • Not Needed
    • Hide

      1. What would you like to communicate to the user about this feature?
      2. Would you like the user to see examples of the syntax and/or executable code and its output?
      3. Which versions of the driver/connector does this apply to?

      Show
      1. What would you like to communicate to the user about this feature? 2. Would you like the user to see examples of the syntax and/or executable code and its output? 3. Which versions of the driver/connector does this apply to?
    • None
    • None
    • None
    • None
    • None
    • None

      Description

      We hit this in production when a replica set node ran out of memory (OOM) and restarted. The client kept failing server selection and only recovered after the client process was restarted. We verified that the bug described below was the cause.

      If a server's own hello response removes it from the topology, _update_servers() closes that server from inside its own monitor task. AsyncPeriodicExecutor.close() then cancels the current task:

      AsyncPeriodicExecutor._run -> Monitor._run -> Topology.on_change -> _process_change
        -> _update_servers -> Server.close -> Monitor.close -> AsyncPeriodicExecutor.close
        -> self._task.cancel()
      

      CancelledError is raised at the next await (in Pool._reset), which has two effects:

      • self._servers.pop(address) never runs, so the old Server stays in Topology._servers.
      • The monitor task ends cancelled, and AsyncPeriodicExecutor.open() will not restart it.

      When the primary's hello lists the address again, _update_servers() reuses the old Server. Its monitor never runs again, so the server stays Unknown with no error. Once every member has gone through this, the client cannot recover without a restart.

      Self-removal reproduced with a standalone response, a mongos response, a me mismatch and a setName mismatch. The synchronous MongoClient recovered in the same standalone scenario.

      Steps to reproduce

      Run with pip install pymongo==4.19.0. No MongoDB server is needed.

      import asyncio, struct
      
      import bson
      from pymongo import AsyncMongoClient
      
      HOSTS = ["127.0.0.1:27101", "127.0.0.1:27102"]
      role = {27101: "secondary", 27102: "primary"}
      
      
      def hello(port):
          doc = {"ok": 1, "minWireVersion": 0, "maxWireVersion": 21}
          if role[port] == "standalone":
              return doc
          return doc | {"setName": "rs", "hosts": HOSTS, "me": f"127.0.0.1:{port}",
                        "primary": HOSTS[1], "secondary": role[port] == "secondary",
                        "isWritablePrimary": role[port] == "primary"}
      
      
      def server(port):
          async def handle(reader, writer):
              try:
                  while True:
                      length, req_id, _, _ = struct.unpack("<iiii", await reader.readexactly(16))
                      body = await reader.readexactly(length - 16)
                      cmd = next(iter(bson.decode(body[5:]))).lower()
                      reply = hello(port) if cmd in ("hello", "ismaster") else {"ok": 1}
                      data = b"\0" * 5 + bson.encode(reply)
                      writer.write(struct.pack("<iiii", 16 + len(data), 0, req_id, 2013) + data)
              except (asyncio.IncompleteReadError, ConnectionError):
                  writer.close()
          return asyncio.start_server(handle, "127.0.0.1", port)
      
      
      async def main():
          servers = [await server(p) for p in role]
          client = AsyncMongoClient(",".join(HOSTS), replicaSet="rs", heartbeatFrequencyMS=500,
                                    serverMonitoringMode="poll", serverSelectionTimeoutMS=2000)
          await client.admin.command("ping")
          for port, original in role.items():  # each node briefly removes itself, then recovers
              role[port] = "standalone"
              await asyncio.sleep(2)
              role[port] = original
              await asyncio.sleep(2)
          try:
              await client.admin.command("ping")
              print("recovered")
          except Exception as exc:
              print(f"{type(exc).__name__}: {exc}")
      
      
      asyncio.run(main())
      

      Expected: prints recovered.

      Actual:

      ServerSelectionTimeoutError: No replica set members found yet, Timeout: 2.0s, Topology Description: <TopologyDescription id: ..., topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('127.0.0.1', 27101) server_type: Unknown, rtt: None>]>
      

      Suggested fix

      Do not cancel the current task in AsyncPeriodicExecutor.close(). _stopped still ends the loop:

      if self._task is not None and self._task is not asyncio.current_task():
          self._task.cancel()
      

      With this change, applied as a monkeypatch, the script prints recovered on 4.19.0 and main.

      Popping the server before await server.close() in _update_servers() also recovered on 4.19.0, but the task still cancels itself, so it only works as extra hardening.

            Assignee:
            Steve Silvester
            Reporter:
            gzpan123 N/A (EXT)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: