Uploaded image for project: 'Python Driver'
  1. Python Driver
  2. PYTHON-607

Gevent 1.0 causes pymongo.MongoReplicaSetClient to hang on epoll call

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 2.7
    • Affects Version/s: 2.6.3
    • Component/s: None
    • Labels:
    • Environment:

      Note from jesse: This is arguably a bug in Gevent, or in how people use Gevent. Now that Gevent 1.0 uses a thread pool for DNS resolution by default, one must be careful not to import a module that does network I/O when it is imported. There are a number of solid workarounds:

      • Don't do create a MongoClient or MongoReplicaSetClient at the time your module is imported. Instead, create the client in your main program file.
      • Or, set the GEVENT_RESOLVER environment variable to "ares".
      • Or, execute unicode('foo').encode('idna') before importing a module that creates a client at import time.
      • Or, use Gevent 0.13.8 instead of 1.0 until Gevent's issue 349 is resolved.

      Original bug report:

      Today I did a deployment of an application without pinning the virtualenv to gevent 0.13, which caused the latest version of gevent 1.0 to install into the virtualenv. I found that I'm seeing the instantiation of the MongoReplicaSetClient cause the program to hang. Running an strace on the PID, I see that the process is halted at an epoll_wait() syscall.

      # strace -p 3615
      Process 3615 attached - interrupt to quit
      clock_gettime(CLOCK_MONOTONIC, {8908174, 879475984}) = 0
      gettimeofday({1386196654, 512281}, NULL) = 0
      clock_gettime(CLOCK_MONOTONIC, {8908174, 879932439}) = 0

      That file descriptor is an anonymous inode, according to lsof:

      python  4913 webapp    4u  0000      0,9        0   4738 anon_inode

      The traceback after interrupting the process is here:

      $ python mcal-cron.py
      Traceback (most recent call last):
        File "mcal-cron.py", line 28, in <module>
          from mcal.config import config
        File "/home/webapp/apps/mcal/mcal/mcal/__init__.py", line 19, in <module>
          mongo = pymongo.MongoReplicaSetClient(config.MONGO_IP, use_greenlets=False, **config.MONGO_KWARGS)
        File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/pymongo/mongo_replica_set_client.py", line 742, in __init__
        File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/pymongo/mongo_replica_set_client.py", line 1179, in refresh
          res, connection_pool, ping_time = self.__is_master(host)
        File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/pymongo/mongo_replica_set_client.py", line 1056, in __is_master
          sock_info = connection_pool.get_socket()
        File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/pymongo/pool.py", line 339, in get_socket
          sock_info, from_pool = self.connect(pair), False
        File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/pymongo/pool.py", line 265, in connect
          sock = self.create_connection(pair)
        File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/pymongo/pool.py", line 237, in create_connection
          for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
        File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/gevent/socket.py", line 621, in getaddrinfo
          return get_hub().resolver.getaddrinfo(host, port, family, socktype, proto, flags)
        File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/gevent/resolver_thread.py", line 34, in getaddrinfo
          return self.pool.apply_e(self.expected_errors, _socket.getaddrinfo, args, kwargs)
        File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/gevent/threadpool.py", line 222, in apply_e
          success, result = self.spawn(wrap_errors, expected_errors, function, args, kwargs).get()
        File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/gevent/event.py", line 226, in get
          result = self.hub.switch()
        File "/home/webapp/apps/mcal/venv/local/lib/python2.7/site-packages/gevent/hub.py", line 331, in switch
          return greenlet.switch(self)

      Rolling back the installed version of gevent to 0.13 resolves the issue. As you can see above, I did explicitly set the use_greenlets arg to False, though since the default is already false, this had no effect.

      Let me know if I can help further.

            jesse@mongodb.com A. Jesse Jiryu Davis
            angstwad Paul Durivage
            2 Vote for this issue
            4 Start watching this issue