Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-20944

Contention on ThreadPoolTaskExecutor::_mutex

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: 3.1.9
    • Fix Version/s: 3.2.0-rc3
    • Component/s: Internal Code
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Sprint:
      Sharding B (10/30/15), QuInt C (11/23/15)

      Description

      The new mongos query path schedules commands to run on the shards using the ThreadPoolTaskExecutor attached to Grid. There is a single instance of the ThreadPoolTaskExecutor per mongos process which synchronizes access to its internal queues through a single mutex. Performance testing has shown that, when there are a sufficient number of concurrent queries executing on mongos, the threads spend much of their time blocked waiting to acquire this lock. Hacking mongos to create a ThreadPoolTaskExecutor per connection thread shows a 3X increase in throughput for a find-by-_id benchRun() workload.

      For reference, I'm using the following script to generate load on mongos:

      #!/bin/bash
       
      THREADS=$1
      BENCH_SECONDS=$2
      RPC=opQueryOnly
      PORT=20004
       
      ./mongo --port $PORT --quiet --eval 'db.foo.insert({_id:1})'
       
      ./mongo --port $PORT --rpcProtocols $RPC --eval '
      ops = [{op:"find", ns:"test.foo", query: {_id: 1}}]
      results = benchRun({ops:ops, parallel:'$THREADS', seconds:'$BENCH_SECONDS', host:"localhost:'$PORT'"})
      print(Math.round(results["totalOps/s"]))
      '
      

        Attachments

        1. vtune_locks_and_waits.png
          227 kB
          David Storch

          Issue Links

            Activity

              People

              Assignee:
              david.storch David Storch
              Reporter:
              david.storch David Storch
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: