Uploaded image for project: 'Python Driver'
  1. Python Driver
  2. PYTHON-2484

Locks in PyMongo should be sanitized on fork

    • Type: Icon: Improvement Improvement
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 4.3
    • Affects Version/s: None
    • Component/s: None
    • Labels:

      Forking a multi-threaded Python process is becoming more and more reliable in recent versions of Python. See

      However a MongoClient is still not fork-safe (also neither is ObjectId because it contains a global lock): https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe

      We should investigate if we can use os.register_at_fork (new in Python 3.7) to allow a MongoClient to be fork-safe.

      For example, something like this could work:

      def _after_fork():
          for client in _CLIENTS:
              client._after_fork()  # Reset all state after a fork, (reallocate locks, clear server session pool, clear connection pool, etc...)
          # Reset any other Locks used in PyMongo 
          ObjectId._after_fork()
      
      if hasattr(os, "register_at_fork"):
          os.register_at_fork(after_in_child=_after_fork)
      

      The above change would make pymongo mostly fork-safe in Python 3.7+. I added the mostly caveat because the python interpreter itself still has some fork deadlock issues due to various locks in the standard library (see the above python bugs).

        1. image.png
          4 kB
          Ben Warner

            Assignee:
            ben.warner@mongodb.com Ben Warner (Inactive)
            Reporter:
            alexander.golin@mongodb.com Alexander Golin (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: