Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-35219

Regain MongoDB balancer performance with sessions

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: 3.6.5
    • Fix Version/s: 3.6.12, 4.0.8, 4.1.10
    • Component/s: Sharding
    • Labels:
    • Backwards Compatibility:
      Fully Compatible
    • Backport Requested:
      v4.0, v3.6
    • Sprint:
      Sharding 2018-07-16, Sharding 2018-12-17, Sharding 2018-12-31, Sharding 2019-01-14, Sharding 2019-01-28, Sharding 2019-02-11, Sharding 2019-02-25, Sharding 2019-03-11, Sharding 2019-03-25
    • Case:
    • Linked BF Score:
      0

      Description

       Original Summary

      MongoDB 3.6 Balancer much slower than 3.4

      Original Description

      I have a test for balancer between 3.6 and 3.4, and I found balancing on 3.6 is much slower than 3.4.

      I insert 100 million docs to 3.6 and 3.4, the single doc size is 2 kbytes. 

      Collection initial as below:

      db.runCommand({shardCollection: "ycsb.test", key: {_id: "hashed"}, numInitialChunks: 6500}) 

      Test Results:

      1. 3.4.15 with MMAPv1 engine: 
        1. from 1 shard to 2 shards: use 41866 seconds, after balance, 3250 chunks on shard.
        2. from 2 sahrds to 4 shards: use 30630 seconds, after balance 1625 chunks on 1 shard.
      2. 3.6.5 with MMAPv1 engine:
        1. from 1 shard to 2 shards: use 90200 seconds, after balance, 3250 chunks on shard.
        2. from 2 sahrds to 4 shards: use 44679 seconds, after balance 1625 chunks on 1 shard.
      3. 3.4.15 with wiredTiger engine:
        1. from 1 shard to 2 shards: use 35635 seconds, after balance, 3250 chunks on shard.
        2. from 2 sahrds to 4 shards: use 10740 seconds, after balance 1625 chunks on 1 shard.
      4. 3.6.5 with wiredTiger engine:
        1. from 1 shard to 2 shards: use 49762 seconds, after balance, 3250 chunks on shard.
        2. from 2 sahrds to 4 shards: use 18961 seconds, after balance 1625 chunks on 1 shard.

      MongoDB configuration for MMAPv1 engine:

      security:
        authorization: disabled
      sharding:
        clusterRole: shardsvr
      replication:
        replSetName: rs1
      systemLog:
        logAppend: true
        destination: file
        path: /home/adun/3.4/log/mongod.log

      processManagement:
        fork: true
        pidFilePath: /home/adun/3.4/log/mongod.pid

      net:
        port: 27017
        bindIp: 127.0.0.1,192.168.10.31
        maxIncomingConnections: 65536

      storage:
        dbPath: /home/adun/3.4/data
        directoryPerDB: true
        engine: mmapv1

      MongoDB configuration for wiredTiger engine:

      security:
        authorization: disabled
      sharding:
        clusterRole: shardsvr
      replication:
        replSetName: rs1
      systemLog:
        logAppend: true
        destination: file
        path: /home/adun/3.4/log/mongod.log

      processManagement:
        fork: true
        pidFilePath: /home/adun/3.4/log/mongod.pid

      net:
        port: 27017
        bindIp: 127.0.0.1,192.168.10.31
        maxIncomingConnections: 65536

      storage:
        dbPath: /home/adun/3.4/data
        directoryPerDB: true
        engine: wiredTiger

        wiredTiger:
          engineConfig:
            cacheSizeGB: 32
            directoryForIndexes: true

      Other Questions:

      1. If we set numInitialChunks to a small value, such as 100, mongodb will create/split chunk by itself, but even if we insert the same data/same records number, 3.6 will create chunks abont 10% more than 3.4. (both MMAPv1 and wiredTiger)

        Attachments

        1. chart.png
          chart.png
          11 kB
        2. chart-2.png
          chart-2.png
          9 kB

          Issue Links

            Activity

              People

              • Votes:
                2 Vote for this issue
                Watchers:
                29 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: