Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-949

format heap-use-after-free

    • Type: Icon: Task Task
    • Resolution: Done
    • WT2.2
    • Affects Version/s: None
    • Component/s: None

      @michaelcahill, I've stared at this one, but I don't see it.

      One thread is looking at page->type:

          733         parent_decr += size;
          734         WT_TRET(__wt_session_fotxn_add(session, pindex, size));
          735         switch (parent->type) {
          736         case WT_PAGE_ROW_INT:
          737         case WT_PAGE_ROW_LEAF:
          738                 if ((ikey = __wt_ref_key_instantiated(ref)) == NULL)
          739                         break;
          740                 size = sizeof(WT_IKEY) + ikey->size;
          741                 parent_decr += size;
          742                 WT_TRET(__wt_session_fotxn_add(session, ikey, size));
          743                 break;
          744         }
      

      and the page has been evicted/freed.

      I want to blame the code that loops until we get a matching ref->home and parent, but I don't see any problems there.

      We looked at page->type 20 lines before, line 713, so this must have happened simultaneously, so somehow our parent got evicted while we were splitting into it.

      Is there some way an internal page could be selected for eviction after a split but before the child pages appear on it? I don't see how that's possible, either.

      =================================================================
      ==105189==ERROR: AddressSanitizer: heap-use-after-free on address 0x607000bc18f0 at pc 0x4febbf bp 0x7f64b0cdbc50 sp 0x7f64b0cdbc48
      READ of size 1 at 0x607000bc18f0 thread T79
          #0 0x4febbe in __wt_split_evict <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/rec_split.c>:735
          WT-1 0x4fb3e3 in __rec_page_dirty_update <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/rec_evict.c>:172
          WT-2 0x4fb3e3 in __wt_rec_evict <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/rec_evict.c>:88
          WT-3 0x4d1bdd in __wt_evict_page <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_evict.c>:393
          WT-4 0x4d1bdd in __wt_evict_lru_page <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_evict.c>:977
          WT-5 0x599585 in __wt_cache_full_check <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/include/cache.i>:93
          WT-6 0x599585 in __cursor_enter <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/include/cursor.i>:57
          WT-7 0x599585 in __curfile_enter <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/include/cursor.i>:94
          WT-8 0x599585 in __cursor_func_init <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/include/cursor.i>:141
          WT-9 0x59cc17 in __wt_btcur_update <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_cursor.c>:522:38
          WT-10 0x547062 in __curfile_update <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/cursor/cur_file.c>:262
          WT-11 0x452fd7 in row_update <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/../../../test/format/ops.c>:701
          WT-12 0x452fd7 in ops <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/../../../test/format/ops.c>:372
          WT-13 0x4415b3 in __asan::AsanThread::ThreadStart(unsigned long) (<http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/t+0x4415b3)>
          WT-14 0x339e807c52 in start_thread (/lib64/libpthread.so.0+0x339e807c52)
          WT-15 0x339e0f5dbc in __clone (/lib64/libc.so.6+0x339e0f5dbc)
      0x607000bc18f0 is located 64 bytes inside of 72-byte region [0x607000bc18b0,0x607000bc18f8)
      freed by thread T66 here:
          #0 0x43af54 in free (<http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/t+0x43af54)>
          WT-1 0x5a50f3 in __wt_page_out <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_discard.c>:97
          WT-2 0x4fb738 in __rec_page_dirty_update <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/rec_evict.c>:192
          WT-3 0x4fb738 in __wt_rec_evict <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/rec_evict.c>:88
          WT-4 0x4d1bdd in __wt_evict_page <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_evict.c>:393
          WT-5 0x4d1bdd in __wt_evict_lru_page <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_evict.c>:977
      previously allocated by thread T78 here:
          #0 0x43b0a3 in calloc (<http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/t+0x43b0a3)>
          WT-1 0x4889cc in __wt_calloc <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/os_posix/os_alloc.c>:38
          WT-2 0x4db993 in __wt_page_inmem <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_page.c>:294
          WT-3 0x4debe6 in __wt_cache_read <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_read.c>:109
          WT-4 0x4da997 in __wt_page_in_func <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_page.c>:46
          WT-5 0x52c51e in __wt_page_swap_func <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/include/btree.i>:704
          WT-6 0x52c51e in __wt_row_search <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/row_srch.c>:278
          WT-7 0x59b713 in __cursor_row_search <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_cursor.c>:158
          WT-8 0x59b713 in __wt_btcur_insert <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_cursor.c>:393
          WT-9 0x5464c7 in __curfile_insert <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/cursor/cur_file.c>:223
      Thread T79 created by T0 here:
          #0 0x436fa0 in __interceptor_pthread_create (<http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/t+0x436fa0)>
          WT-1 0x450949 in wts_ops <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/../../../test/format/ops.c>:94
          WT-2 0x456992 in main <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/../../../test/format/t.c>:180
      Thread T66 created by T0 here:
          #0 0x436fa0 in __interceptor_pthread_create (<http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/t+0x436fa0)>
          WT-1 0x450949 in wts_ops <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/../../../test/format/ops.c>:94
          WT-2 0x456992 in main <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/../../../test/format/t.c>:180
      Thread T78 created by T0 here:
          #0 0x436fa0 in __interceptor_pthread_create (<http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/t+0x436fa0)>
          WT-1 0x450949 in wts_ops <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/../../../test/format/ops.c>:94
          WT-2 0x456992 in main <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/../../../test/format/t.c>:180
      SUMMARY: AddressSanitizer: heap-use-after-free <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/rec_split.c>:735 __wt_split_evict
      Shadow bytes around the buggy address:
        0x0c0e801702c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0c0e801702d0: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fd fd
        0x0c0e801702e0: fd fd fd fd fd fd fd fa fa fa fa fa fd fd fd fd
        0x0c0e801702f0: fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa
        0x0c0e80170300: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
      =>0x0c0e80170310: fd fa fa fa fa fa fd fd fd fd fd fd fd fd[fd]fa
        0x0c0e80170320: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0c0e80170330: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0c0e80170340: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fd fd
        0x0c0e80170350: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd
        0x0c0e80170360: fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa
      Shadow byte legend (one shadow byte represents 8 application bytes):
        Addressable:           00
        Partially addressable: 01 02 03 04 05 06 07
        Heap left redzone:     fa
        Heap right redzone:    fb
        Freed heap region:     fd
        Stack left redzone:    f1
        Stack mid redzone:     f2
        Stack right redzone:   f3
        Stack partial redzone: f4
        Stack after return:    f5
        Stack use after scope: f8
        Global redzone:        f9
        Global init order:     f6
        Poisoned by user:      f7
        ASan internal:         fe
      ==105189==ABORTING
      + status=1
      + '[' -f RUNDIR/run ']'
      + cat RUNDIR/run
      ############################################
      #  RUN PARAMETERS
      ############################################
      auto_throttle=1
      firstfit=1
      # bitcnt not applicable to this run
      bloom=1
      bloom_bit_count=48
      bloom_hash_count=7
      bloom_oldest=0
      cache=3
      checkpoints=1
      checksum=off
      chunk_size=7
      compaction=0
      compression=none
      data_extend=0
      data_source=file
      delete_pct=12
      dictionary=0
      file_type=row-store
      hot_backups=0
      huffman_key=0
      huffman_value=0
      insert_pct=18
      internal_key_truncation=1
      internal_page_max=14
      key_gap=1
      key_max=72
      key_min=25
      leaf_page_max=10
      merge_max=19
      merge_threads=1
      mmap=1
      ops=100000
      prefix_compression=0
      prefix_compression_min=8
      repeat_data_pct=7
      reverse=0
      rows=100000
      runs=1
      split_pct=58
      statistics=0
      threads=30
      value_max=814
      value_min=19
      # wiredtiger_config not applicable to this run
      write_pct=59
      ############################################
      

            Assignee:
            michael.cahill@mongodb.com Michael Cahill (Inactive)
            Reporter:
            keith.bostic@mongodb.com Keith Bostic (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

              Created:
              Updated:
              Resolved: