Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-20303

Negative scaling at low thread count under WiredTiger when inserting large documents

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical - P2
    • Resolution: Fixed
    • Affects Version/s: 3.0.6, 3.1.7
    • Fix Version/s: 3.0.8, 3.2.0-rc0
    • Component/s: WiredTiger
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Backport Completed:

      Description

      • test inserts 100 kB documents into a single initially empty collection from multiple client threads
      • 24 cpus (12 cores * 2 hyperthreads), 64 GB memory, SSD storage
      • standalone mongod (so no oplog); no journal; checkpoints disabled to reduce variability
      • mongod restarted between each test for repeatability

      Following graph shows 5 successive runs (with mongod restart between each run) at 1, 2, 4, 8 and 16 client threads:

      • aggregate throughput declines as number of client threads increases
      • disk is mostly idle and becomes more so as op rate declines
      • cpu is mostly idle at all thread counts
      • eviction activity increases as client threads increase
      • yet bytes in cache also grows as thread count increases - eviction is somehow unable to keep up with the (increasingly meager) insertion rate?
      • memory allocations increase dramatically in spite of smaller insertion rate
      • threads appear to be blocked trying to acquire pages, probably accounting for lower insertion rate

      Following gdbmon profile showing two consecutive runs with 2 and 16 threads respectively:

      In the second half of the timeline, showing the run with 16 client threads,

      • A - much of the time many or most of the 16 threads are waiting (sleeping)
      • B - while one thread is evicting

        Attachments

        1. 3.0.0.png
          3.0.0.png
          30 kB
        2. 3.0.6.png
          3.0.6.png
          31 kB
        3. 3.1.0.png
          3.1.0.png
          32 kB
        4. 3.1.7.png
          3.1.7.png
          33 kB
        5. gdbmon.html
          326 kB
        6. gdbmon.log
          924 kB
        7. gdbmon.png
          gdbmon.png
          280 kB
        8. iostat.log
          233 kB
        9. latest.png
          latest.png
          33 kB
        10. ss.log
          2.90 MB
        11. stats.html
          1.30 MB
        12. stats.png
          stats.png
          276 kB

          Issue Links

            Activity

              People

              • Votes:
                1 Vote for this issue
                Watchers:
                18 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: