-
Type:
Bug
-
Resolution: Done
-
Priority:
Major - P3
-
Affects Version/s: 2.6.3
-
Component/s: None
-
Labels:None
-
Environment:$ uname -r
3.2.0-51-virtual
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 12.04.2 LTS
Release: 12.04
Codename: precise
$ python -V
Python 2.7.3
$ pip freeze
Flask==0.10.1
Flask-WTF==0.9.3
Jinja2==2.7.1
MarkupSafe==0.18
SOAPpy==0.12.5
WTForms==1.0.5
Werkzeug==0.9.4
argparse==1.2.1
configparser==3.3.0r2
docutils==0.11
fpconst==0.7.2
gevent==1.0
greenlet==0.4.1
gunicorn==18.0
itsdangerous==0.23
pymongo==2.6.3
python-dateutil==2.2
pytz==2013.8
requests==2.0.1
rspy==0.1
six==1.4.1
wsgiref==0.1.2
wstools==0.4.3$ uname -r 3.2.0-51-virtual $ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 12.04.2 LTS Release: 12.04 Codename: precise $ python -V Python 2.7.3 $ pip freeze Flask==0.10.1 Flask-WTF==0.9.3 Jinja2==2.7.1 MarkupSafe==0.18 SOAPpy==0.12.5 WTForms==1.0.5 Werkzeug==0.9.4 argparse==1.2.1 configparser==3.3.0r2 docutils==0.11 fpconst==0.7.2 gevent==1.0 greenlet==0.4.1 gunicorn==18.0 itsdangerous==0.23 pymongo==2.6.3 python-dateutil==2.2 pytz==2013.8 requests==2.0.1 rspy==0.1 six==1.4.1 wsgiref==0.1.2 wstools==0.4.3
Note from jesse: This is arguably a bug in Gevent, or in how people use Gevent. Now that Gevent 1.0 uses a thread pool for DNS resolution by default, one must be careful not to import a module that does network I/O when it is imported. There are a number of solid workarounds:
- Don't do create a MongoClient or MongoReplicaSetClient at the time your module is imported. Instead, create the client in your main program file.
- Or, set the GEVENT_RESOLVER environment variable to "ares".
- Or, execute unicode('foo').encode('idna') before importing a module that creates a client at import time.
- Or, use Gevent 0.13.8 instead of 1.0 until Gevent's issue 349 is resolved.
Original bug report:
Today I did a deployment of an application without pinning the virtualenv to gevent 0.13, which caused the latest version of gevent 1.0 to install into the virtualenv. I found that I'm seeing the instantiation of the MongoReplicaSetClient cause the program to hang. Running an strace on the PID, I see that the process is halted at an epoll_wait() syscall.
# strace -p 3615
Process 3615 attached - interrupt to quit
clock_gettime(CLOCK_MONOTONIC, {8908174, 879475984}) = 0
gettimeofday({1386196654, 512281}, NULL) = 0
clock_gettime(CLOCK_MONOTONIC, {8908174, 879932439}) = 0
epoll_wait(4,
That file descriptor is an anonymous inode, according to lsof:
python 4913 webapp 4u 0000 0,9 0 4738 anon_inode
The traceback after interrupting the process is here:
$ python mcal-cron.py
^CKeyboardInterrupt
Traceback (most recent call last):
File "mcal-cron.py", line 28, in <module>
from mcal.config import config
File "/home/webapp/apps/mcal/mcal/mcal/__init__.py", line 19, in <module>
mongo = pymongo.MongoReplicaSetClient(config.MONGO_IP, use_greenlets=False, **config.MONGO_KWARGS)
File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/pymongo/mongo_replica_set_client.py", line 742, in __init__
self.refresh()
File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/pymongo/mongo_replica_set_client.py", line 1179, in refresh
res, connection_pool, ping_time = self.__is_master(host)
File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/pymongo/mongo_replica_set_client.py", line 1056, in __is_master
sock_info = connection_pool.get_socket()
File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/pymongo/pool.py", line 339, in get_socket
sock_info, from_pool = self.connect(pair), False
File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/pymongo/pool.py", line 265, in connect
sock = self.create_connection(pair)
File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/pymongo/pool.py", line 237, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/gevent/socket.py", line 621, in getaddrinfo
return get_hub().resolver.getaddrinfo(host, port, family, socktype, proto, flags)
File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/gevent/resolver_thread.py", line 34, in getaddrinfo
return self.pool.apply_e(self.expected_errors, _socket.getaddrinfo, args, kwargs)
File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/gevent/threadpool.py", line 222, in apply_e
success, result = self.spawn(wrap_errors, expected_errors, function, args, kwargs).get()
File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/gevent/event.py", line 226, in get
result = self.hub.switch()
File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/gevent/hub.py", line 331, in switch
return greenlet.switch(self)
KeyboardInterrupt
Rolling back the installed version of gevent to 0.13 resolves the issue. As you can see above, I did explicitly set the use_greenlets arg to False, though since the default is already false, this had no effect.
Let me know if I can help further.