Summary

Under gevent monkey-patching, if greenlets are killed (GreenletExit) while blocked on PyMongo's internal locks/conditions – which real gevent applications do constantly, e.g. every websocket disconnect kills its handler greenlet mid-operation – the synchronous client can enter a permanent, unrecoverable deadlock: every subsequent operation in the process blocks forever acquiring Topology.lock / Pool.lock, including the driver's own server monitors. No client timeout fires, because the stuck acquires are untimed (with self._lock:) and sit _before the code where serverSelectionTimeoutMS / waitQueueTimeoutMS are consulted. Only killing the process recovers.

This is a regression introduced with the 4.9 restructure: 4.8.0 does not exhibit the bug under an identical trigger, while 4.10.1 and 4.18.0 (latest at time of filing) both deadlock reproducibly in ~1-2 minutes with the attached script – unmodified libraries, natural timing.

We first hit this in production at Drivers4Me (drivers4me.com – driver-on-demand platform; Flask monolith on gunicorn GeventWebSocketWorker, MongoDB Atlas): roughly weekly, one worker process would wedge with all Mongo operations frozen, and – as collateral – every request greenlet that had taken a SQL connection before its Mongo call pinned it forever, exhausting the SQL pool too. We captured the wedged process live (greenlet stack dump via gdb injection) before finding the deterministic reproduction.

Environment

Steps to reproduce

Attached: pymongo_gevent_deadlock_repro.py – self-contained, ~250 lines. It runs N greenlets doing find_one/insert_one against one shared client with a small pool, plus a "reaper" that kills and respawns a random worker greenlet every few milliseconds (simulating websocket disconnects). A watchdog detects when the op counter freezes while workers are alive, dumps every greenlet's stack (gc walk over gr_frame), and reports whether the parked frames match the driver-lock signature.

docker run -d --rm -p 27017:27017 mongo:7
pip install "pymongo==4.18.0" "gevent==24.10.1"
REAPER_MS=20 WORKERS=300 MAX_POOL=3 STALL_S=15 \
    MONGO_URI=mongodb://localhost:27017 python pymongo_gevent_deadlock_repro.py

Expected: operations keep completing (killed workers are replaced; individual op failures are fine).
Actual: the op counter freezes permanently with all workers alive; the dump shows essentially every greenlet blocked on PyMongo's internal locks. Exit code 1.

Observed results across versions (same machine, same parameters):

The script also has an optional AMPLIFY_RACE=1 mode that widens gevent's courtesy-yield window (synthetic, clearly labeled); with it both 4.10.1 and 4.18.0 seize in <=40 s. All natural-timing results above are with unmodified libraries.

The deadlocked state (dumps)

Production, PyMongo 4.10.1 – 760 suspended greenlets in the wedged process:

• 520 blocked at pymongo/synchronous/topology.py:282 -> select_servers -> with self._lock: (every application Mongo op)
• 10 blocked at pymongo/synchronous/pool.py:1439 -> _get_conn -> {
  Unknown macro: {with self.lock}

  }

• 2 server-monitor greenlets blocked at topology.py:537 -> on_change -> {
  Unknown macro: {with self._lock}

  }

• exactly one PyMongo greenlet not blocked acquiring a lock – suspended mid-notify() while holding driver locks:

<application websocket handler>
  -> Collection.find_one()
  -> ... -> pool.checkout.__exit__ -> Pool.checkin      (pool.py:1593)
  -> self._max_connecting_cond.notify()
  -> threading.py:376 notify -> threading.py:289 _is_owned
  -> self._lock.acquire(False)                           <- non-blocking, guaranteed to fail
  -> gevent/thread.py:286 acquire -> sleep()             <- gevent yields on this path by design
  -> gevent/hub.py:159 sleep -> waiter.get()             <- suspended here, locks still held

This state persisted for hours (verified by repeated dumps) until the process was killed.

Reproducer, PyMongo 4.18.0 – same seizure, shifted internals: 300/305 suspended greenlets parked in threading.Condition.wait -> waiter.acquire / gevent BoundedSemaphore.acquire under PyMongo frames, plus the periodic-executor greenlets. The package layout changed between 4.10 and 4.18 (frames no longer under synchronous/), but the deadlock is unchanged.

Analysis

Two deterministic ingredients plus one race:

1. PyMongo >=4.9 calls Condition.notify() inside its locked sections on the hot path – e.g. Pool.checkin() -> _max_connecting_cond.notify() runs at the end of every operation while driver locks are held. (4.8.0's pre-restructure pool does not have this shape, which we believe is why it is immune.)
2. CPython's Condition.is_owned() performs a non-blocking acquire(False) that is guaranteed to fail when the caller owns the lock – and gevent's patched Lock.acquire(blocking=False) deliberately performs a cooperative sleep() on the failed path (gevent/thread.py, gevent issue #1464 – added so spin-locks make progress). Net effect: _every PyMongo operation cooperatively yields its greenlet at a point where driver locks are held. This is observable deterministically; it is the exposure, not yet the bug.
3. The race: when greenlets are killed (GreenletExit) while queued on those locks/conditions – as happens whenever a gevent server tears down a connection whose handler is inside a Mongo call – a wakeup is eventually lost in the interaction between kill delivery and gevent's semaphore waiter bookkeeping, and from that point every acquire of Topology._lock/Pool.lock blocks forever. We have not pinned the exact lost-wakeup line (it is in the gevent/CPython interaction), but the trigger is unambiguous: with the reaper on, >=4.9 seizes within minutes; without it, 19M ops ran clean; 4.8.0 survives the identical trigger.

Nothing can recover the process afterward: the acquires are untimed, so serverSelectionTimeoutMS, socketTimeoutMS, connectTimeoutMS and waitQueueTimeoutMS never start counting (the greenlets never reach the code paths those bound), and the topology monitors are themselves deadlocked.

Impact

• Whole-client, whole-process permanent outage of all Mongo operations; in mixed-datastore apps it cascades (greenlets holding other resources – SQL pool slots – freeze with it).
• Affects the documented gevent integration path (monkey.patch_all before imports, per the PyMongo/MongoDB gevent integration docs) with no warning, no error, and no timeout – the failure mode is silence.
• Trigger frequency scales with connection churn; long-lived websocket servers are the worst case.

Workaround

Pin pymongo==4.8.0 (last pre-restructure release). Verified immune to the exact trigger. Bounding timeouts does not help (see Analysis).

Suggested directions

• Avoid Condition.notify() while holding driver locks on the hot path, or use explicitly owned/timed primitives whose ownership check doesn't require a failing non-blocking acquire.
• Harden lock/condition state against GreenletExit/BaseException delivered inside acquire/wait (PyMongo had analogous fixes for gevent.Timeout in the 3.x era, e.g. the changelog entries "using gevent.Timeout to timeout an operation could lead to a deadlock").
• At minimum: a documented statement of (non-)support for the sync client under gevent kill-churn, so users can pin/architect accordingly.

We're happy to run candidate patches through the reproducer and through the production-shaped load harness that found this.