-
Type:
Bug
-
Resolution: Done
-
Priority:
Critical - P2
-
Affects Version/s: 2.2
-
Component/s: None
-
None
mod_wsgi 2.x, Python <= 2.6, apache prefork and worker, PyMongo >= 2.2 with auto_start_request True. Any other combination won't reproduce this behavior. Unfortunately this is a common combination.
mod_wsgi reuses threads for multiple requests with the WSGIDaemonProcess directive. mod_wsgi 2.x, which is still widely deployed in, for example, Ubuntu 10.04, deletes thread state between requests even while it reuses threads, which causes strange behavior in PyMongo 2.2: it leaks one socket per two HTTP requests, resulting in unbounded growth in the number of sockets.
PyMongo <= 2.1.1 puts a socket in a threadlocal variable; this socket is cleaned up (its sock_dealloc is called) when mod_wsgi 2.8 deletes the thread state between HTTP requests. PyMongo 2.2, however, puts a Python SocketInfo object into a thread local; this object's _del_ then chooses whether to return the socket to the pool or to close it; somehow under mod_wsgi 2.8, every second such SocketInfo is never deleted.
Workarounds until this bug is understood and fixed in PyMongo (which will be ASAP):
*) Pass auto_start_request=False to socket() or ReplicaSetsocket(). This avoids using thread locals and thus avoids mod_wsgi 2.8's strange handling of thread state.
*) Upgrade mod_wsgi to the current version 3.2, which preserves thread state across requests.
*) Upgrade Python to 2.7 (this requires recompiling mod_wsgi and setting WSGIPythonHome in your Apache configuration to point to Python 2.7's installation directory) – Python 2.7 sticks an object into the interpreter's thread state and uses a weakref callback to proactively clean up thread local state when a thread dies, so it doesn't leak connections with PyMongo 2.2
*) Downgrade PyMongo to version 2.1.1
For more on differences in thread state handling between mod_wsgi 2 and 3 see:
https://groups.google.com/d/topic/modwsgi/sIs91L9XVgs/discussion
http://code.google.com/p/modwsgi/issues/detail?id=120
Here's my current understanding of PyMongo, Python, and mod_wsgi:
| Python | mod_wsgi | PyMongo | Behavior |
|---|---|---|---|
| <=2.6 | <=2.8 | <=2.1.1 | One new socket per HTTP request, no leaks: use end_request to reuse sockets |
| <=2.6 | <=2.8 | 2.2 | This bug: unbounded socket growth with auto_start_request=True; use end_request or auto_start_request=False |
| <=2.6 | >=3.0 | <=2.1.1 | Each thread reuses same socket for all HTTP requests; no leaks |
| <=2.6 | >=3.0 | 2.2 | Each thread reuses same socket for all HTTP requests; with auto_start_request=False threads safely share sockets |
| >=2.7 | <=2.8 | <=2.1.1 | One new socket per HTTP request, no leaks: use end_request to reuse sockets |
| >=2.7 | <=2.8 | 2.2 | Threads get a socket per HTTP request, sockets returned to pool and reused as if calling end_request |
| >=2.7 | >=3.0 | <=2.1.1 | One new socket per HTTP request, no leaks: use end_request to reuse sockets |
| >=2.7 | >=3.0 | 2.2 | Each thread reuses same socket for all HTTP requests; with auto_start_request=False threads safely share sockets |
- related to
-
-
- Closed
-