Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-75353

Mongodb v5.0.3 scaling bottleneck/issue

XMLWordPrintableJSON

    • Product Performance
    • ALL
    • Hide

      Use ycsb client,
       
      $ python2.7 ./bin/ycsb load mongodb -jvm-args="-Dlogback.configurationFile=../logback.xml" -s -P workloads/workloadc -threads 20 -p mongodb.url=mongodb://localhost/ycsb

      {{}}

      $ python2.7 ./bin/ycsb run mongodb -jvm-args="-Dlogback.configurationFile=../logback.xml" -s -P workloads/workloadc -threads 80 -p mongodb.url=mongodb://localhost/ycsb

      {}No. records: 40M
      Type of Request: Read-Only queries

      Show
      Use ycsb client,   $ python2.7 ./bin/ycsb load mongodb -jvm-args="-Dlogback.configurationFile=../logback.xml" -s -P workloads/workloadc -threads 20 -p mongodb.url=mongodb://localhost/ycsb {{}} $ python2.7 ./bin/ycsb run mongodb -jvm-args="-Dlogback.configurationFile=../logback.xml" -s -P workloads/workloadc -threads 80 -p mongodb.url=mongodb://localhost/ycsb { } No. records: 40M Type of Request: Read-Only queries

      Hi,

      I've been studying the mongodb scaling impact on a high core count machines. I've observed that the server doesn't scale linearly once it crosses 20 threads. This was tested against Mongodb v5.0.3 and YCSB client (readonly tests). Here is my observation,

      Clients/Threads 1 2 4 8 10 20 30 40 80
       Throughput per thread           1.00          0.99          0.98          0.94          0.94          0.80          0.59          0.43          0.19

      First row is self-explanatory. The second row show per thread throughput over single thread throughput.  Also see the attachment showing the graph. In order to understand the issue, I collected some off-cpu data events. This data showed a lock/pause is the cause of scaling issue in two functions, viz. 1) mongo::ServiceContext::makeOperationContext() and 2) mongo::ServiceContext::__delistOperation(). I've also attached flamegraph showing these bottlenecks.

      Unfortunately, I can't contribute to the project with actual code-contribution due to restrictions on my side but I can submit the issues hence this report. I'm hoping someone from the mongodb community will take a look at this issue and provide a solution.

      I hope this helps as a a good starting point for further analysis.

        1. offcpu.80t.svg
          108 kB
        2. mongodb-thread-scaling.png
          mongodb-thread-scaling.png
          112 kB
        3. mongodb-scaling-ipc.png
          mongodb-scaling-ipc.png
          100 kB
        4. mongodb-scaling-cpu-util.png
          mongodb-scaling-cpu-util.png
          138 kB

            Assignee:
            backlog-server-perf [DO NOT USE] Backlog - Performance Team
            Reporter:
            uttampc NOVALUE Uttam
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated: