Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-3637

Fix a heap use after free from evicting of a page that just split.

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.2.21, 3.4.17, 3.6.0-rc4, WT3.0.0
    • Component/s: None
    • Labels:
    • Sprint:
      Storage 2017-11-13
    • Backport Requested:
      v3.6, v3.4, v3.2

      Description

      The configuration was column-store, but there is not (yet) strong evidence that the failure is column-store specific.

      http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/17291/console

      ==32018==ERROR: AddressSanitizer: heap-use-after-free on address 0x604000110a10 at pc 0x0000007c9089 bp 0x7fe3e6a07850 sp 0x7fe3e6a07848
      WRITE of size 8 at 0x604000110a10 thread T28
          #0 0x7c9088 in __wt_page_inmem /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_page.c:233:13
          #1 0x7dc19d in __page_read /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_read.c:370:2
          #2 0x7d9899 in __wt_page_in_func /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_read.c:506:4
          #3 0x869be7 in __wt_page_swap_func /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/include/btree.i:1429:8
          #4 0x867b3d in __wt_col_search /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/col_srch.c:194:14
          #5 0xa03883 in __cursor_col_search /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_cursor.c:336:2
          #6 0xa0242a in __wt_btcur_search /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_cursor.c:479:3
          #7 0xa0a68f in __wt_btcur_modify /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_cursor.c:1251:2
          #8 0x8db542 in __curfile_modify /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/cursor/cur_file.c:303:2
          #9 0x52b704 in col_modify /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/ops.c:1344:16
          #10 0x526795 in ops /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/ops.c:769:11
          #11 0x7fe406867dc4 in start_thread (/lib64/libpthread.so.0+0x7dc4)
          #12 0x7fe405a4b76c in __clone (/lib64/libc.so.6+0xf776c)
       
      0x604000110a10 is located 0 bytes inside of 48-byte region [0x604000110a10,0x604000110a40)
      freed by thread T1 here:
          #0 0x4df570 in __interceptor_free /home/bostic/src/llvm40/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:47
          #1 0x608329 in __wt_free_int /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/os_common/os_alloc.c:327:2
          #2 0x79f909 in __wt_free_ref /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_discard.c:331:2
          #3 0x79ffea in __wt_free_ref_index /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_discard.c:348:3
          #4 0x7a1bfe in __free_page_int /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_discard.c:268:2
          #5 0x79ee1c in __page_out_int /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_discard.c:143:3
          #6 0x79e0be in __wt_ref_out_int /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_discard.c:60:2
          #7 0x79f15e in __wt_ref_out /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_discard.c:70:2
          #8 0x5cf89e in __evict_page_dirty_update /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_page.c:378:5
          #9 0x5cc720 in __wt_evict /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_page.c:184:3
          #10 0x5b229f in __evict_page /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_lru.c:2242:2
          #11 0x5ac9d7 in __evict_lru_pages /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_lru.c:1149:14
          #12 0x5abe65 in __wt_evict_thread_run /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_lru.c:318:3
          #13 0x7297c3 in __thread_run /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/support/thread_group.c:31:3
          #14 0x7fe406867dc4 in start_thread (/lib64/libpthread.so.0+0x7dc4)
       
      previously allocated by thread T2 here:
          #0 0x4dfa6d in calloc /home/bostic/src/llvm40/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:74
          #1 0x606b82 in __wt_calloc /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/os_common/os_alloc.c:52:11
          #2 0x811934 in __wt_multi_to_ref /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_split.c:1593:2
          #3 0x82d510 in __split_multi /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_split.c:2092:3
          #4 0x8143c4 in __split_multi_lock /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_split.c:2139:13
          #5 0x814259 in __wt_split_multi /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_split.c:2167:2
          #6 0x5cf3b5 in __evict_page_dirty_update /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_page.c:341:4
          #7 0x5cc720 in __wt_evict /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_page.c:184:3
          #8 0x5b229f in __evict_page /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_lru.c:2242:2
          #9 0x5ac9d7 in __evict_lru_pages /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_lru.c:1149:14
          #10 0x5abe65 in __wt_evict_thread_run /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_lru.c:318:3
          #11 0x7297c3 in __thread_run /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/support/thread_group.c:31:3
          #12 0x7fe406867dc4 in start_thread (/lib64/libpthread.so.0+0x7dc4)
       
      Thread T28 created by T0 here:
          #0 0x4373c1 in __interceptor_pthread_create /home/bostic/src/llvm40/projects/compiler-rt/lib/asan/asan_interceptors.cc:305
          #1 0x61d2c0 in __wt_thread_create /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/os_posix/os_thread.c:30:2
          #2 0x5224cb in wts_ops /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/ops.c:164:3
          #3 0x532fcd in main /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/t.c:209:5
          #4 0x7fe405975b34 in __libc_start_main (/lib64/libc.so.6+0x21b34)
       
      Thread T1 created by T0 here:
          #0 0x4373c1 in __interceptor_pthread_create /home/bostic/src/llvm40/projects/compiler-rt/lib/asan/asan_interceptors.cc:305
          #1 0x61d2c0 in __wt_thread_create /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/os_posix/os_thread.c:30:2
          #2 0x7273d5 in __thread_group_resize /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/support/thread_group.c:213:3
          #3 0x727df0 in __wt_thread_group_create /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/support/thread_group.c:304:2
          #4 0x5add95 in __wt_evict_create /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_lru.c:477:2
          #5 0x5773ec in __wt_connection_workers /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/conn/conn_open.c:238:2
          #6 0x551fb0 in wiredtiger_open /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/conn/conn_api.c:2627:2
          #7 0x53d05b in wts_open /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/wts.c:273:2
          #8 0x532e90 in main /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/t.c:190:3
          #9 0x7fe405975b34 in __libc_start_main (/lib64/libc.so.6+0x21b34)
       
      Thread T2 created by T0 here:
          #0 0x4373c1 in __interceptor_pthread_create /home/bostic/src/llvm40/projects/compiler-rt/lib/asan/asan_interceptors.cc:305
          #1 0x61d2c0 in __wt_thread_create /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/os_posix/os_thread.c:30:2
          #2 0x7273d5 in __thread_group_resize /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/support/thread_group.c:213:3
          #3 0x727df0 in __wt_thread_group_create /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/support/thread_group.c:304:2
          #4 0x5add95 in __wt_evict_create /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_lru.c:477:2
          #5 0x5773ec in __wt_connection_workers /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/conn/conn_open.c:238:2
          #6 0x551fb0 in wiredtiger_open /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/conn/conn_api.c:2627:2
          #7 0x53d05b in wts_open /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/wts.c:273:2
          #8 0x532e90 in main /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/t.c:190:3
          #9 0x7fe405975b34 in __libc_start_main (/lib64/libc.so.6+0x21b34)
       
      SUMMARY: AddressSanitizer: heap-use-after-free /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_page.c:233:13 in __wt_page_inmem
      Shadow bytes around the buggy address:
        0x0c088001a0f0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fd
        0x0c088001a100: fa fa 00 00 00 00 00 00 fa fa 00 00 00 00 00 fa
        0x0c088001a110: fa fa 00 00 00 00 00 00 fa fa fa fa fa fa fa fa
        0x0c088001a120: fa fa 00 00 00 00 00 00 fa fa 00 00 00 00 00 fa
        0x0c088001a130: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa
      =>0x0c088001a140: fa fa[fd]fd fd fd fd fd fa fa 00 00 00 00 00 00
        0x0c088001a150: fa fa fa fa fa fa fa fa fa fa 00 00 00 00 00 00
        0x0c088001a160: fa fa fd fd fd fd fd fd fa fa fa fa fa fa fa fa
        0x0c088001a170: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa
        0x0c088001a180: fa fa fa fa fa fa fa fa fa fa 00 00 00 00 00 00
        0x0c088001a190: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
      Shadow byte legend (one shadow byte represents 8 application bytes):
        Addressable:           00
        Partially addressable: 01 02 03 04 05 06 07 
        Heap left redzone:       fa
        Freed heap region:       fd
        Stack left redzone:      f1
        Stack mid redzone:       f2
        Stack right redzone:     f3
        Stack after return:      f5
        Stack use after scope:   f8
        Global redzone:          f9
        Global init order:       f6
        Poisoned by user:        f7
        Container overflow:      fc
        Array cookie:            ac
        Intra object redzone:    bb
        ASan internal:           fe
        Left alloca redzone:     ca
        Right alloca redzone:    cb
      ==32018==ABORTING
      

      Here's the CONFIG:

      ############################################
      #  RUN PARAMETERS
      ############################################
      abort=0
      alter=0
      auto_throttle=1
      backups=0
      bitcnt=5
      bloom=1
      bloom_bit_count=9
      bloom_hash_count=12
      bloom_oldest=0
      cache=36
      checkpoints=1
      checksum=uncompressed
      chunk_size=8
      compaction=0
      compression=lz4
      data_extend=0
      data_source=table
      delete_pct=7
      dictionary=0
      direct_io=0
      encryption=rotn-7
      evict_max=5
      file_type=variable-length column-store
      firstfit=0
      huffman_key=0
      huffman_value=0
      independent_thread_rng=1
      in_memory=0
      insert_pct=12
      internal_key_truncation=1
      internal_page_max=17
      isolation=random
      key_gap=3
      key_max=32
      key_min=10
      leaf_page_max=16
      leak_memory=0
      logging=0
      logging_archive=1
      logging_compression=none
      logging_prealloc=0
      long_running_txn=0
      lsm_worker_threads=3
      merge_max=14
      mmap=1
      modify_pct=39
      ops=100000
      prefix_compression=0
      prefix_compression_min=7
      quiet=1
      read_pct=23
      rebalance=1
      repeat_data_pct=60
      reverse=0
      rows=100000
      runs=1
      salvage=1
      split_pct=58
      statistics=0
      statistics_server=0
      threads=9
      timer=360
      transaction_timestamps=0
      transaction-frequency=45
      value_max=3975
      value_min=15
      verify=1
      wiredtiger_config=
      write_pct=19
      ############################################
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                vamsi.krishna Vamsi Boyapati
                Reporter:
                keith.bostic Keith Bostic
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: