Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-4171

Enabling tree walk timing stress causes excessive slowdown

    • Type: Icon: Improvement Improvement
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 3.6.9, 4.0.2, 4.1.2, WT3.2.0
    • Affects Version/s: None
    • Component/s: None
    • Labels:
    • Storage Non-NYC 2018-07-16, Storage Engines 2018-07-30

      There have been a few very long running test/format jobs recently, a common configuration is:

      timing_stress_split_9=1
      

      I attached in a debugger and the job is here:

      (gdb) where
      #0  0x00007f50cfc367a3 in select () from /lib64/libc.so.6
      #1  0x000000000045b147 in __wt_sleep (seconds=0, micro_seconds=13340) at ../src/os_posix/os_sleep.c:32
      #2  0x000000000050deb8 in __wt_timing_stress (session=0x7f50d13281b0, flag=1024)
          at ../src/include/misc.i:272
      #3  0x0000000000510886 in __wt_row_search (session=0x7f50d13281b0, srch_key=0x2248f50, leaf=0x0, 
          cbt=0x2248e50, insert=false, restore=false) at ../src/btree/row_srch.c:435
      #4  0x00000000005965c3 in __cursor_row_search (session=0x7f50d13281b0, cbt=0x2248e50, leaf=0x0, 
          insert=false) at ../src/btree/bt_cursor.c:385
      #5  0x0000000000596c2e in __wt_btcur_search (cbt=0x2248e50) at ../src/btree/bt_cursor.c:530
      #6  0x000000000052f07c in __curfile_search (cursor=0x2248e50) at ../src/cursor/cur_file.c:200
      #7  0x000000000040a07c in read_row_worker (cursor=0x2248e50, keyno=613197, key=0x7ffc6730f190, 
          value=0x7ffc6730f160, sn=false) at ../../../test/format/ops.c:1252
      #8  0x0000000000409eee in wts_read_scan () at ../../../test/format/ops.c:1202
      #9  0x000000000040d89c in main (argc=4, argv=0x7ffc6730f398) at ../../../test/format/t.c:207
      (gdb) f 8
      #8  0x0000000000409eee in wts_read_scan () at ../../../test/format/ops.c:1202
      1202			switch (ret = read_row_worker(
      (gdb) p keyno
      $14 = 613197
      (gdb) p g.key_cnt
      $11 = 1000000
      

      After about 21 hours.

      The long running job showing characteristics above:
      http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress/67017/

            Assignee:
            keith.bostic@mongodb.com Keith Bostic (Inactive)
            Reporter:
            alexander.gorrod@mongodb.com Alexander Gorrod
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: