Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-20303

Negative scaling at low thread count under WiredTiger when inserting large documents

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Critical - P2 Critical - P2
    • 3.0.8, 3.2.0-rc0
    • Affects Version/s: 3.0.6, 3.1.7
    • Component/s: WiredTiger
    • Labels:
      None
    • Fully Compatible
    • ALL

      • test inserts 100 kB documents into a single initially empty collection from multiple client threads
      • 24 cpus (12 cores * 2 hyperthreads), 64 GB memory, SSD storage
      • standalone mongod (so no oplog); no journal; checkpoints disabled to reduce variability
      • mongod restarted between each test for repeatability

      Following graph shows 5 successive runs (with mongod restart between each run) at 1, 2, 4, 8 and 16 client threads:

      • aggregate throughput declines as number of client threads increases
      • disk is mostly idle and becomes more so as op rate declines
      • cpu is mostly idle at all thread counts
      • eviction activity increases as client threads increase
      • yet bytes in cache also grows as thread count increases - eviction is somehow unable to keep up with the (increasingly meager) insertion rate?
      • memory allocations increase dramatically in spite of smaller insertion rate
      • threads appear to be blocked trying to acquire pages, probably accounting for lower insertion rate

      Following gdbmon profile showing two consecutive runs with 2 and 16 threads respectively:

      In the second half of the timeline, showing the run with 16 client threads,

      • A - much of the time many or most of the 16 threads are waiting (sleeping)
      • B - while one thread is evicting

        1. stats.png
          stats.png
          276 kB
        2. stats.html
          1.30 MB
        3. ss.log
          2.90 MB
        4. latest.png
          latest.png
          33 kB
        5. iostat.log
          233 kB
        6. gdbmon.png
          gdbmon.png
          280 kB
        7. gdbmon.log
          924 kB
        8. gdbmon.html
          326 kB
        9. 3.1.7.png
          3.1.7.png
          33 kB
        10. 3.1.0.png
          3.1.0.png
          32 kB
        11. 3.0.6.png
          3.0.6.png
          31 kB
        12. 3.0.0.png
          3.0.0.png
          30 kB

            Assignee:
            keith.bostic@mongodb.com Keith Bostic (Inactive)
            Reporter:
            bruce.lucas@mongodb.com Bruce Lucas (Inactive)
            Votes:
            1 Vote for this issue
            Watchers:
            17 Start watching this issue

              Created:
              Updated:
              Resolved: