Description
Forking a multi-threaded Python process is becoming more and more reliable in recent versions of Python. See
- https://bugs.python.org/issue6721
- https://bugs.python.org/issue25920
- https://bugs.python.org/issue40089
However a MongoClient is still not fork-safe (also neither is ObjectId because it contains a global lock): https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe
We should investigate if we can use os.register_at_fork (new in Python 3.7) to allow a MongoClient to be fork-safe.
For example, something like this could work:
def _after_fork(): |
for client in _CLIENTS: |
client._after_fork() # Reset all state after a fork, (reallocate locks, clear server session pool, clear connection pool, etc...) |
# Reset any other Locks used in PyMongo |
ObjectId._after_fork()
|
|
|
if hasattr(os, "register_at_fork"): |
os.register_at_fork(after_in_child=_after_fork) |
The above change would make pymongo mostly fork-safe in Python 3.7+. I added the mostly caveat because the python interpreter itself still has some fork deadlock issues due to various locks in the standard library (see the above python bugs).
Attachments
Issue Links
- is related to
-
PYTHON-1745 Raise an error if an opened MongoClient is used after a fork
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
PYTHON-3393 Add stress test for MongoClient fork safety
-
- Closed
-
- related to
-
PYTHON-3257 Facing " connection pool paused" issue after upgrading pymongo from 3.12.3 to python 4.1.0
-
- Closed
-
-
-
- Closed
-
- links to