Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-5518

Split-parent code can race with other threads when checking the WT_REF.state

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • WT10.0.0, 4.4.0-rc0, 4.7.0
    • Affects Version/s: None
    • Component/s: None
    • None
    • 8
    • Storage Engines 2020-02-24
    • v4.4, v4.2, v4.0

      Summary:
      Fix another race revealed by WT-5219 adding more locking of WT_REFs in the system.


      test/format failed on develop on margay-aws, http://build.wiredtiger.com:8080/job/wiredtiger-test-race-condition-stress-sanitizer/35320
      with a heap-buffer-overflow in the split code:

      =================================================================
      ==19000==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x61e0009dde60 at pc 0x0000009fdf2f bp 0x7f75a91f2e90 sp 0x7f75a91f2e88
      WRITE of size 8 at 0x61e0009dde60 thread T5
          #0 0x9fdf2e in __split_parent /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/../src/btree/bt_split.c:700:27
          #1 0x9f2b9a in __split_reverse /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/../src/btree/bt_split.c:2161:11
          #2 0x9f29e2 in __wt_split_reverse /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/../src/btree/bt_split.c:2181:5
          #3 0x652add in __evict_delete_ref /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/../src/evict/evict_page.c:272:24
          #4 0x64fe7b in __evict_page_dirty_update /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/../src/evict/evict_page.c:357:9
          #5 0x64abc7 in __wt_evict /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/../src/evict/evict_page.c:192:9
          #6 0x62b504 in __evict_page /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/../src/evict/evict_lru.c:2238:5
          #7 0x624380 in __evict_lru_pages /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/../src/evict/evict_lru.c:1106:20
          #8 0x623380 in __wt_evict_thread_run /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/../src/evict/evict_lru.c:311:9
          #9 0x851ac1 in __thread_run /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/../src/support/thread_group.c:31:9
          #10 0x7f75afc276da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
          #11 0x7f75aed5988e in clone /build/glibc-OTsEL5/glibc-2.27/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95
      
      0x61e0009dde60 is located 0 bytes to the right of 2528-byte region [0x61e0009dd480,0x61e0009dde60)
      allocated by thread T5 here:
          #0 0x4c7a7a in calloc (/mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/test/format/t+0x4c7a7a)
          #1 0x6e066a in __wt_calloc /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/../src/os_common/os_alloc.c:50:14
          #2 0x9fd941 in __split_parent /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/../src/btree/bt_split.c:685:5
          #3 0x9f2b9a in __split_reverse /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/../src/btree/bt_split.c:2161:11
          #4 0x9f29e2 in __wt_split_reverse /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/../src/btree/bt_split.c:2181:5
          #5 0x652add in __evict_delete_ref /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/../src/evict/evict_page.c:272:24
          #6 0x64fe7b in __evict_page_dirty_update /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/../src/evict/evict_page.c:357:9
          #7 0x64abc7 in __wt_evict /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/../src/evict/evict_page.c:192:9
          #8 0x62b504 in __evict_page /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/../src/evict/evict_lru.c:2238:5
          #9 0x624380 in __evict_lru_pages /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/../src/evict/evict_lru.c:1106:20
          #10 0x623380 in __wt_evict_thread_run /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/../src/evict/evict_lru.c:311:9
          #11 0x851ac1 in __thread_run /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/../src/support/thread_group.c:31:9
          #12 0x7f75afc276da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
      
      Thread T5 created by T0 here:
          #0 0x4afe5d in pthread_create (/mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/test/format/t+0x4afe5d)
          #1 0x700645 in __wt_thread_create /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/../src/os_posix/os_thread.c:28:5
          #2 0x84f0e4 in __thread_group_resize /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/../src/support/thread_group.c:201:9
          #3 0x84fc23 in __wt_thread_group_create /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/../src/support/thread_group.c:285:5
          #4 0x62598e in __wt_evict_create /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/../src/evict/evict_lru.c:477:5
          #5 0x5c9971 in __wt_connection_workers /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/../src/conn/conn_open.c:227:5
          #6 0x589f80 in wiredtiger_open /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/../src/conn/conn_api.c:2675:5
          #7 0x52eb16 in wts_open /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/test/format/../../../test/format/wts.c:277:5
          #8 0x521fc4 in main /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/test/format/../../../test/format/t.c:270:9
          #9 0x7f75aec59b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
      
      SUMMARY: AddressSanitizer: heap-buffer-overflow /mnt/data0/jenkins/workspace/wiredtiger-test-race-condition-stress-sanitizer/build_posix/../src/btree/bt_split.c:700:27 in __split_parent
      

            Assignee:
            keith.bostic@mongodb.com Keith Bostic (Inactive)
            Reporter:
            sue.loverso@mongodb.com Susan LoVerso
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: