Uploaded image for project: 'Python Driver'
  1. Python Driver
  2. PYTHON-1276

Tailable cursor consumes a lot of CPU but only with local connections

    • Type: Icon: Bug Bug
    • Resolution: Works as Designed
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Environment:
      Tested only with PyMongo 3.4.0, Python 2.7.6 and Python 3.4.3 on Ubuntu 14.04 x64. Also tested with PyMongo 3.4.0, and Python 2.7 on OSX
      MongoDB server version tested: 3.0.6, 3.2.8 and 3.4.0

      I built a command line tool called Mongotail on top of PyMongo that allows you to see the logs that MongoDB's profiler saves in the capped collection system.profile. When is executed with the option -f Mongotail connects with a tailable cursor and works as expected.

      If you connect with a remote MongoDB server like this:

      $ mongotail myremoteip/mydatabase -f
      

      It takes around the the 1-3% of the CPU usage, that is quite acceptable.

      But if you connect with a local MongoDB server ( mongotail localhost/mydatabase -f ), always takes 40% of the CPU with a modest computer, or around 15-20% in a more fast computer, like my notebook with a i7 processor and a SSD drive.

      You can see the source code of the project, the connection is made like this:

      cursor = db.system.profile.find(QUERY, projection=fields)
      #...
      cursor.add_option(2) # Set the tailable flag
      while cursor.alive:
          result = next(cursor)
          print_obj(result, ...)  # Print the result in the console
      

      I profiled the application to find where is the problem, and iI found that the higher CPU usage is made in a function called _receive_data_on_socket from the network module of PyMongo, but I don't know why this only happens with local connection.

      This is the profile result analysed with KCacheGrind:

            Assignee:
            shane.harvey@mongodb.com Shane Harvey
            Reporter:
            mrsarm Mariano Ruiz
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: