Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-20303

Negative scaling at low thread count under WiredTiger when inserting large documents



    • Type: Bug
    • Status: Closed
    • Priority: Critical - P2
    • Resolution: Fixed
    • Affects Version/s: 3.0.6, 3.1.7
    • Fix Version/s: 3.0.8, 3.2.0-rc0
    • Component/s: WiredTiger
    • Labels:
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
    • Backport Completed:


      • test inserts 100 kB documents into a single initially empty collection from multiple client threads
      • 24 cpus (12 cores * 2 hyperthreads), 64 GB memory, SSD storage
      • standalone mongod (so no oplog); no journal; checkpoints disabled to reduce variability
      • mongod restarted between each test for repeatability

      Following graph shows 5 successive runs (with mongod restart between each run) at 1, 2, 4, 8 and 16 client threads:

      • aggregate throughput declines as number of client threads increases
      • disk is mostly idle and becomes more so as op rate declines
      • cpu is mostly idle at all thread counts
      • eviction activity increases as client threads increase
      • yet bytes in cache also grows as thread count increases - eviction is somehow unable to keep up with the (increasingly meager) insertion rate?
      • memory allocations increase dramatically in spite of smaller insertion rate
      • threads appear to be blocked trying to acquire pages, probably accounting for lower insertion rate

      Following gdbmon profile showing two consecutive runs with 2 and 16 threads respectively:

      In the second half of the timeline, showing the run with 16 client threads,

      • A - much of the time many or most of the 16 threads are waiting (sleeping)
      • B - while one thread is evicting


        1. 3.0.0.png
          30 kB
          Bruce Lucas
        2. 3.0.6.png
          31 kB
          Bruce Lucas
        3. 3.1.0.png
          32 kB
          Bruce Lucas
        4. 3.1.7.png
          33 kB
          Bruce Lucas
        5. gdbmon.html
          326 kB
          Bruce Lucas
        6. gdbmon.log
          924 kB
          Bruce Lucas
        7. gdbmon.png
          280 kB
          Bruce Lucas
        8. iostat.log
          233 kB
          Bruce Lucas
        9. latest.png
          33 kB
          Bruce Lucas
        10. ss.log
          2.90 MB
          Bruce Lucas
        11. stats.html
          1.30 MB
          Bruce Lucas
        12. stats.png
          276 kB
          Bruce Lucas

          Issue Links



              keith.bostic Keith Bostic
              bruce.lucas Bruce Lucas
              1 Vote for this issue
              17 Start watching this issue