Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-2894

Create workload that shows negative scaling when overwhelmed

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: WT2.9.0, 3.2.10, 3.3.14
    • Labels:
      None
    • # Replies:
      16
    • Last comment by Customer:
      true

      Description

      There have been a few MongoDB workloads that cause a cascading performance degradation in WiredTiger. The application is attempting to push through more workload than the system can handle, as application threads become co-opted into eviction the throughput rate reduces further.

      When the throughput rate drops, the application opens more threads to the database, attempting to keep the amount of throughput stable. As the number of threads grows, WiredTiger successfully completes less and less work.

      The bottle neck appears to be eviction. We should create a workload in standalone WiredTiger that replicates the negative scaling.

      1. 2894.tar.gz
        263 kB
        Sue LoVerso
      1. w2894_recent_stalls.png
        82 kB
      2. wt2894_326.png
        69 kB
      3. wt2894_329.png
        77 kB
      4. wt2894_recent.png
        86 kB

        Issue Links

          Activity

          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: WT-2894 Add wtperf stress workload that tries to induce negative scaling (#3034)

          Via use a high thread count and throttled operations.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/f82cf084e733f745ef437192fbde1235b69509dc

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: WT-2894 Add wtperf stress workload that tries to induce negative scaling (#3034) Via use a high thread count and throttled operations. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/f82cf084e733f745ef437192fbde1235b69509dc
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: WT-2894 Add wtperf stress workload that tries to induce negative scaling (#3034)

          Via use a high thread count and throttled operations.
          Branch: mongodb-3.4
          https://github.com/wiredtiger/wiredtiger/commit/f82cf084e733f745ef437192fbde1235b69509dc

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: WT-2894 Add wtperf stress workload that tries to induce negative scaling (#3034) Via use a high thread count and throttled operations. Branch: mongodb-3.4 https://github.com/wiredtiger/wiredtiger/commit/f82cf084e733f745ef437192fbde1235b69509dc
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: Import wiredtiger: 5bc03723a7e77c96b1d5e45a97173654872c727a from branch mongodb-3.4

          ref: deeb0f589a..5bc03723a7
          for: 3.3.13

          WT-2859 Improve statistics macros to not collect stats when statistics=none is set
          WT-2894 Create workload that shows negative scaling when overwhelmed
          WT-2900 Add ARM8 build support to WiredTiger and fix ARM CRC assembler tags
          WT-2902 Enhance eviction to work better with update heavy YCSB load
          WT-2905 dead code: MongoDB Coverity #99881
          WT-2908 Add dry-run support to python unit-test
          WT-2910 test_inmem01 can abort due to stuck cache
          WT-2911 add support for gcc6
          WT-2912 make --enable-strict run on the zseries test box
          WT-2913 stuck cache after latest merge
          WT-2914 Fix link error on OS/X for checksum_init call
          WT-2916 whitespace check not working
          Branch: master
          https://github.com/mongodb/mongo/commit/7693fa59c4470db729d85af99fb9cc3b264fa8c8

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: Import wiredtiger: 5bc03723a7e77c96b1d5e45a97173654872c727a from branch mongodb-3.4 ref: deeb0f589a..5bc03723a7 for: 3.3.13 WT-2859 Improve statistics macros to not collect stats when statistics=none is set WT-2894 Create workload that shows negative scaling when overwhelmed WT-2900 Add ARM8 build support to WiredTiger and fix ARM CRC assembler tags WT-2902 Enhance eviction to work better with update heavy YCSB load WT-2905 dead code: MongoDB Coverity #99881 WT-2908 Add dry-run support to python unit-test WT-2910 test_inmem01 can abort due to stuck cache WT-2911 add support for gcc6 WT-2912 make --enable-strict run on the zseries test box WT-2913 stuck cache after latest merge WT-2914 Fix link error on OS/X for checksum_init call WT-2916 whitespace check not working Branch: master https://github.com/mongodb/mongo/commit/7693fa59c4470db729d85af99fb9cc3b264fa8c8
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: WT-2894 Add wtperf stress workload that tries to induce negative scaling (#3034)

          Via use a high thread count and throttled operations.
          Branch: mongodb-3.2
          https://github.com/wiredtiger/wiredtiger/commit/f82cf084e733f745ef437192fbde1235b69509dc

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: WT-2894 Add wtperf stress workload that tries to induce negative scaling (#3034) Via use a high thread count and throttled operations. Branch: mongodb-3.2 https://github.com/wiredtiger/wiredtiger/commit/f82cf084e733f745ef437192fbde1235b69509dc
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: Import wiredtiger: bb18c439152379fb528af217345be40819b01fc4 from branch mongodb-3.2

          ref: 911c940ada..bb18c43915
          for: 3.2.10

          WT-2859 Improve statistics macros to not collect stats when statistics=none is set
          WT-2894 Create workload that shows negative scaling when overwhelmed
          WT-2900 Add ARM8 build support to WiredTiger and fix ARM CRC assembler tags
          WT-2902 Enhance eviction to work better with update heavy YCSB load
          WT-2905 dead code: MongoDB Coverity #99881
          WT-2908 Add dry-run support to python unit-test
          WT-2910 test_inmem01 can abort due to stuck cache
          WT-2911 add support for gcc6
          WT-2912 make --enable-strict run on the zseries test box
          WT-2913 stuck cache after latest merge
          WT-2914 Fix link error on OS/X for checksum_init call
          WT-2916 whitespace check not working
          Branch: v3.2
          https://github.com/mongodb/mongo/commit/dfca65ea3a32b1e0c832ff7ee2d7b667819ec64d

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: Import wiredtiger: bb18c439152379fb528af217345be40819b01fc4 from branch mongodb-3.2 ref: 911c940ada..bb18c43915 for: 3.2.10 WT-2859 Improve statistics macros to not collect stats when statistics=none is set WT-2894 Create workload that shows negative scaling when overwhelmed WT-2900 Add ARM8 build support to WiredTiger and fix ARM CRC assembler tags WT-2902 Enhance eviction to work better with update heavy YCSB load WT-2905 dead code: MongoDB Coverity #99881 WT-2908 Add dry-run support to python unit-test WT-2910 test_inmem01 can abort due to stuck cache WT-2911 add support for gcc6 WT-2912 make --enable-strict run on the zseries test box WT-2913 stuck cache after latest merge WT-2914 Fix link error on OS/X for checksum_init call WT-2916 whitespace check not working Branch: v3.2 https://github.com/mongodb/mongo/commit/dfca65ea3a32b1e0c832ff7ee2d7b667819ec64d

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since reply:
                35 weeks, 4 days ago
                Date of 1st Reply: