Critical - P2
mod_wsgi 2.x, Python <= 2.6, apache prefork and worker, PyMongo >= 2.2 with auto_start_request True. Any other combination won't reproduce this behavior. Unfortunately this is a common combination.
mod_wsgi reuses threads for multiple requests with the WSGIDaemonProcess directive. mod_wsgi 2.x, which is still widely deployed in, for example, Ubuntu 10.04, deletes thread state between requests even while it reuses threads, which causes strange behavior in PyMongo 2.2: it leaks one socket per two HTTP requests, resulting in unbounded growth in the number of sockets.
PyMongo <= 2.1.1 puts a socket in a threadlocal variable; this socket is cleaned up (its sock_dealloc is called) when mod_wsgi 2.8 deletes the thread state between HTTP requests. PyMongo 2.2, however, puts a Python SocketInfo object into a thread local; this object's _del_ then chooses whether to return the socket to the pool or to close it; somehow under mod_wsgi 2.8, every second such SocketInfo is never deleted.
Workarounds until this bug is understood and fixed in PyMongo (which will be ASAP):
*) Pass auto_start_request=False to socket() or ReplicaSetsocket(). This avoids using thread locals and thus avoids mod_wsgi 2.8's strange handling of thread state.
*) Upgrade mod_wsgi to the current version 3.2, which preserves thread state across requests.
*) Upgrade Python to 2.7 (this requires recompiling mod_wsgi and setting WSGIPythonHome in your Apache configuration to point to Python 2.7's installation directory) – Python 2.7 sticks an object into the interpreter's thread state and uses a weakref callback to proactively clean up thread local state when a thread dies, so it doesn't leak connections with PyMongo 2.2
*) Downgrade PyMongo to version 2.1.1
For more on differences in thread state handling between mod_wsgi 2 and 3 see:
Here's my current understanding of PyMongo, Python, and mod_wsgi:
|<=2.6||<=2.8||<=2.1.1||One new socket per HTTP request, no leaks: use end_request to reuse sockets|
|<=2.6||<=2.8||2.2||This bug: unbounded socket growth with auto_start_request=True; use end_request or auto_start_request=False|
|<=2.6||>=3.0||<=2.1.1||Each thread reuses same socket for all HTTP requests; no leaks|
|<=2.6||>=3.0||2.2||Each thread reuses same socket for all HTTP requests; with auto_start_request=False threads safely share sockets|
|>=2.7||<=2.8||<=2.1.1||One new socket per HTTP request, no leaks: use end_request to reuse sockets|
|>=2.7||<=2.8||2.2||Threads get a socket per HTTP request, sockets returned to pool and reused as if calling end_request|
|>=2.7||>=3.0||<=2.1.1||One new socket per HTTP request, no leaks: use end_request to reuse sockets|
|>=2.7||>=3.0||2.2||Each thread reuses same socket for all HTTP requests; with auto_start_request=False threads safely share sockets|