Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-20303

Negative scaling at low thread count under WiredTiger when inserting large documents

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical - P2
    • Resolution: Fixed
    • Affects Version/s: 3.0.6, 3.1.7
    • Fix Version/s: 3.0.8, 3.2.0-rc0
    • Component/s: WiredTiger
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Backport Completed:

      Description

      • test inserts 100 kB documents into a single initially empty collection from multiple client threads
      • 24 cpus (12 cores * 2 hyperthreads), 64 GB memory, SSD storage
      • standalone mongod (so no oplog); no journal; checkpoints disabled to reduce variability
      • mongod restarted between each test for repeatability

      Following graph shows 5 successive runs (with mongod restart between each run) at 1, 2, 4, 8 and 16 client threads:

      • aggregate throughput declines as number of client threads increases
      • disk is mostly idle and becomes more so as op rate declines
      • cpu is mostly idle at all thread counts
      • eviction activity increases as client threads increase
      • yet bytes in cache also grows as thread count increases - eviction is somehow unable to keep up with the (increasingly meager) insertion rate?
      • memory allocations increase dramatically in spite of smaller insertion rate
      • threads appear to be blocked trying to acquire pages, probably accounting for lower insertion rate

      Following gdbmon profile showing two consecutive runs with 2 and 16 threads respectively:

      In the second half of the timeline, showing the run with 16 client threads,

      • A - much of the time many or most of the 16 threads are waiting (sleeping)
      • B - while one thread is evicting
      1. gdbmon.html
        326 kB
        Bruce Lucas
      2. gdbmon.log
        924 kB
        Bruce Lucas
      3. iostat.log
        233 kB
        Bruce Lucas
      4. ss.log
        2.90 MB
        Bruce Lucas
      5. stats.html
        1.30 MB
        Bruce Lucas
      1. 3.0.0.png
        30 kB
      2. 3.0.6.png
        31 kB
      3. 3.1.0.png
        32 kB
      4. 3.1.7.png
        33 kB
      5. gdbmon.png
        280 kB
      6. latest.png
        33 kB
      7. stats.png
        276 kB

        Issue Links

          Activity

          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}

          Message: Merge pull request #2233 from wiredtiger/SERVER-20303

          SERVER-20303: tuning in-memory splits for when inserting large objects
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/ad56c6a7f9256fd0377d6441425a64df6760e6be

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'} Message: Merge pull request #2233 from wiredtiger/ SERVER-20303 SERVER-20303 : tuning in-memory splits for when inserting large objects Branch: develop https://github.com/wiredtiger/wiredtiger/commit/ad56c6a7f9256fd0377d6441425a64df6760e6be
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}

          Message: Merge pull request #2233 from wiredtiger/SERVER-20303

          SERVER-20303: tuning in-memory splits for when inserting large objects
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/ad56c6a7f9256fd0377d6441425a64df6760e6be

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'} Message: Merge pull request #2233 from wiredtiger/ SERVER-20303 SERVER-20303 : tuning in-memory splits for when inserting large objects Branch: develop https://github.com/wiredtiger/wiredtiger/commit/ad56c6a7f9256fd0377d6441425a64df6760e6be
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}

          Message: SERVER-20303 Require a minimum item count for in-memory splits.

          Merge pull request #2236 from wiredtiger/SERVER-20303-mjc

          (cherry picked from commit f12d478229822ee19b5767e05e73083adbe095f4)
          Branch: mongodb-3.0
          https://github.com/wiredtiger/wiredtiger/commit/01dbcf110a971b8155ad76ff3b2c8b528c1d9b73

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'} Message: SERVER-20303 Require a minimum item count for in-memory splits. Merge pull request #2236 from wiredtiger/ SERVER-20303 -mjc (cherry picked from commit f12d478229822ee19b5767e05e73083adbe095f4) Branch: mongodb-3.0 https://github.com/wiredtiger/wiredtiger/commit/01dbcf110a971b8155ad76ff3b2c8b528c1d9b73
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}

          Message: SERVER-20303 Require a minimum item count for in-memory splits.

          Merge pull request #2236 from wiredtiger/SERVER-20303-mjc

          (cherry picked from commit f12d478229822ee19b5767e05e73083adbe095f4)
          Branch: mongodb-3.0
          https://github.com/wiredtiger/wiredtiger/commit/01dbcf110a971b8155ad76ff3b2c8b528c1d9b73

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'} Message: SERVER-20303 Require a minimum item count for in-memory splits. Merge pull request #2236 from wiredtiger/ SERVER-20303 -mjc (cherry picked from commit f12d478229822ee19b5767e05e73083adbe095f4) Branch: mongodb-3.0 https://github.com/wiredtiger/wiredtiger/commit/01dbcf110a971b8155ad76ff3b2c8b528c1d9b73
          Hide
          pasette Dan Pasette added a comment - - edited

          Verified this fix after the latest WT code drop on a 24 core box.

          Before:
          1 thread : 9319
          2 threads : 11434
          4 threads : 9766
          8 threadss : 4188
          16 threads : 898

          After:
          1 threads : 13566
          2 threads : 24115
          4 threads : 36321
          8 threads : 45561
          16 threads : 45304

          Show
          pasette Dan Pasette added a comment - - edited Verified this fix after the latest WT code drop on a 24 core box. Before: 1 thread : 9319 2 threads : 11434 4 threads : 9766 8 threadss : 4188 16 threads : 898 After: 1 threads : 13566 2 threads : 24115 4 threads : 36321 8 threads : 45561 16 threads : 45304

            People

            • Votes:
              1 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: