Locks in PyMongo should be sanitized on fork

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Fixed
    • Priority: Major - P3
    • 4.3
    • Affects Version/s: None
    • Component/s: None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Forking a multi-threaded Python process is becoming more and more reliable in recent versions of Python. See

      However a MongoClient is still not fork-safe (also neither is ObjectId because it contains a global lock): https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe

      We should investigate if we can use os.register_at_fork (new in Python 3.7) to allow a MongoClient to be fork-safe.

      For example, something like this could work:

      def _after_fork():
    for client in _CLIENTS:
        client._after_fork()  # Reset all state after a fork, (reallocate locks, clear server session pool, clear connection pool, etc...)
    # Reset any other Locks used in PyMongo 
    ObjectId._after_fork()

if hasattr(os, "register_at_fork"):
    os.register_at_fork(after_in_child=_after_fork)

      The above change would make pymongo mostly fork-safe in Python 3.7+. I added the mostly caveat because the python interpreter itself still has some fork deadlock issues due to various locks in the standard library (see the above python bugs).

        1. image.png
          image.png
          4 kB

            Assignee:
            Ben Warner (Inactive)
            Reporter:
            Alexander Golin (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: