Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-2923

heap-use-after-free on address in compaction

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: WT2.9.0, 3.2.10, 3.3.15
    • Labels:
      None

      Description

      heap-use-after-free on address in compaction

      This one is real and reproducible.

      http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer-ppc/1718/

      ./configure CCAS=gcc 'CC=clang -fsanitize=address' 'CFLAGS=-g -fno-omit-frame-pointer -I/usr/lib/gcc/ppc64le-redhat-linux/4.8.2/include' --enable-diagnostic --with-builtins=lz4,snappy,zlib --with-berkeleydb=/home/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer-ppc/build_posix/db
      

      =================================================================
      ==13388==ERROR: AddressSanitizer: heap-use-after-free on address 0x0a1000045a70 at pc 0x0000103f5e04 bp 0x3fff80f8c660 sp 0x3fff80f8c680
      READ of size 8 at 0x0a1000045a70 thread T87
          #0 0x103f5e00 in __wt_ref_info /home/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer-ppc/build_posix/../src/include/btree.i:1070:18
          #1 0x103f580c in __wt_compact_page_skip /home/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer-ppc/build_posix/../src/btree/bt_compact.c:189:2
          #2 0x104c5e44 in __tree_walk_internal /home/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer-ppc/build_posix/../src/btree/bt_walk.c:491:6
          #3 0x104c4714 in __wt_tree_walk /home/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer-ppc/build_posix/../src/btree/bt_walk.c:678:10
          #4 0x103f35ac in __wt_compact /home/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer-ppc/build_posix/../src/btree/bt_compact.c:139:3
          #5 0x10349074 in __compact_file /home/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer-ppc/build_posix/../src/session/session_compact.c:235:3
          #6 0x10346d9c in __wt_session_compact /home/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer-ppc/build_posix/../src/session/session_compact.c:312:4
          #7 0x10120db4 in compact /home/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer-ppc/build_posix/test/format/../../../test/format/compact.c:66:14
          #8 0x100eefe8 in __asan::AsanThread::ThreadStart(unsigned long, __sanitizer::atomic_uintptr_t*) /home/mcahill/src/llvm-3.8.1.src/projects/compiler-rt-3.8.1.src/lib/asan/asan_thread.cc:183
          #9 0x100322e8 in asan_thread_start(void*) /home/mcahill/src/llvm-3.8.1.src/projects/compiler-rt-3.8.1.src/lib/asan/asan_interceptors.cc:228
          #10 0x3fff98008940 in start_thread (/lib64/power8/libpthread.so.0+0x8940)
      

      ############################################
      #  RUN PARAMETERS
      ############################################
      abort=0
      auto_throttle=1
      backups=0
      bitcnt=8
      bloom=1
      bloom_bit_count=8
      bloom_hash_count=8
      bloom_oldest=1
      cache=34
      checkpoints=1
      checksum=uncompressed
      chunk_size=8
      compaction=1
      compression=lz4-noraw
      data_extend=0
      data_source=file
      delete_pct=12
      dictionary=0
      direct_io=0
      encryption=none
      evict_max=5
      file_type=row-store
      firstfit=0
      huffman_key=0
      huffman_value=0
      in_memory=0
      insert_pct=6
      internal_key_truncation=1
      internal_page_max=9
      isolation=random
      key_gap=13
      key_max=110
      key_min=11
      leaf_page_max=16
      leak_memory=0
      logging=1
      logging_archive=1
      logging_compression=lz4
      logging_prealloc=1
      long_running_txn=0
      lsm_worker_threads=4
      merge_max=17
      mmap=1
      ops=100000
      prefix_compression=1
      prefix_compression_min=3
      quiet=1
      repeat_data_pct=50
      reverse=0
      rows=100000
      runs=1
      rebalance=1
      salvage=1
      split_pct=68
      statistics=0
      statistics_server=0
      threads=25
      timer=20
      transaction-frequency=6
      value_max=3265
      value_min=17
      verify=1
      wiredtiger_config=
      write_pct=86
      ############################################
      

        Issue Links

          Activity

          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}

          Message: WT-2923 heap-use-after-free on address in compaction (#3053)

          We can't look at the WT_REF address without some kind of lock as the address can change underneath us. For example, if we take a copy of WT_REF.addr.addr while it's referencing an on-page cell, and then the page is evicted before we crack that address, we'll potentially read freed memory. Lock down the WT_REF before reading the address.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/1c2c9268e122c4da1e49f14b7b17f20183b1a991

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'} Message: WT-2923 heap-use-after-free on address in compaction (#3053) We can't look at the WT_REF address without some kind of lock as the address can change underneath us. For example, if we take a copy of WT_REF.addr.addr while it's referencing an on-page cell, and then the page is evicted before we crack that address, we'll potentially read freed memory. Lock down the WT_REF before reading the address. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/1c2c9268e122c4da1e49f14b7b17f20183b1a991
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}

          Message: WT-2923 heap-use-after-free on address in compaction (#3053)

          We can't look at the WT_REF address without some kind of lock as the address can change underneath us. For example, if we take a copy of WT_REF.addr.addr while it's referencing an on-page cell, and then the page is evicted before we crack that address, we'll potentially read freed memory. Lock down the WT_REF before reading the address.
          Branch: mongodb-3.4
          https://github.com/wiredtiger/wiredtiger/commit/1c2c9268e122c4da1e49f14b7b17f20183b1a991

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'} Message: WT-2923 heap-use-after-free on address in compaction (#3053) We can't look at the WT_REF address without some kind of lock as the address can change underneath us. For example, if we take a copy of WT_REF.addr.addr while it's referencing an on-page cell, and then the page is evicted before we crack that address, we'll potentially read freed memory. Lock down the WT_REF before reading the address. Branch: mongodb-3.4 https://github.com/wiredtiger/wiredtiger/commit/1c2c9268e122c4da1e49f14b7b17f20183b1a991
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}

          Message: WT-2923 heap-use-after-free on address in compaction (#3053)

          We can't look at the WT_REF address without some kind of lock as the address can change underneath us. For example, if we take a copy of WT_REF.addr.addr while it's referencing an on-page cell, and then the page is evicted before we crack that address, we'll potentially read freed memory. Lock down the WT_REF before reading the address.
          Branch: mongodb-3.2
          https://github.com/wiredtiger/wiredtiger/commit/1c2c9268e122c4da1e49f14b7b17f20183b1a991

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'} Message: WT-2923 heap-use-after-free on address in compaction (#3053) We can't look at the WT_REF address without some kind of lock as the address can change underneath us. For example, if we take a copy of WT_REF.addr.addr while it's referencing an on-page cell, and then the page is evicted before we crack that address, we'll potentially read freed memory. Lock down the WT_REF before reading the address. Branch: mongodb-3.2 https://github.com/wiredtiger/wiredtiger/commit/1c2c9268e122c4da1e49f14b7b17f20183b1a991
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'}

          Message: Import wiredtiger: 9cf2f89d6d95e1de797f05ab1fef28695f8bae7b from branch mongodb-3.2

          ref: bb18c43915..9cf2f89d6d
          for: 3.2.10

          WT-2864 Reconfiguring the checkpoint server can lead to hangs
          WT-2874 Change test_compact01 to avoid eviction
          WT-2918 The dist scripts create C files s_whitespace complains about
          WT-2919 Don't mask error returns from style checking scripts
          WT-2921 Reduce the WT_SESSION hazard_size when possible
          WT-2923 heap-use-after-free on address in compaction
          WT-2924 Ensure we are doing eviction when threads are waiting for it
          WT-2925 WT_THREAD_PANIC_FAIL is a WT_THREAD structure flag
          WT-2926 WT_CONNECTION.reconfigure can attempt unlock of not-locked lock
          WT-2928 Eviction failing to switch queues can lead to starvation
          Branch: v3.2
          https://github.com/mongodb/mongo/commit/79d9b3ab5ce20f51c272b4411202710a082d0317

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'} Message: Import wiredtiger: 9cf2f89d6d95e1de797f05ab1fef28695f8bae7b from branch mongodb-3.2 ref: bb18c43915..9cf2f89d6d for: 3.2.10 WT-2864 Reconfiguring the checkpoint server can lead to hangs WT-2874 Change test_compact01 to avoid eviction WT-2918 The dist scripts create C files s_whitespace complains about WT-2919 Don't mask error returns from style checking scripts WT-2921 Reduce the WT_SESSION hazard_size when possible WT-2923 heap-use-after-free on address in compaction WT-2924 Ensure we are doing eviction when threads are waiting for it WT-2925 WT_THREAD_PANIC_FAIL is a WT_THREAD structure flag WT-2926 WT_CONNECTION.reconfigure can attempt unlock of not-locked lock WT-2928 Eviction failing to switch queues can lead to starvation Branch: v3.2 https://github.com/mongodb/mongo/commit/79d9b3ab5ce20f51c272b4411202710a082d0317
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'}

          Message: Import wiredtiger: fc0e7abe82595e579573d42448632f7b36a2d154 from branch mongodb-3.4

          ref: 5bc03723a7..fc0e7abe82
          for: 3.3.15

          WT-2864 Reconfiguring the checkpoint server can lead to hangs
          WT-2874 Change test_compact01 to avoid eviction
          WT-2918 The dist scripts create C files s_whitespace complains about
          WT-2919 Don't mask error returns from style checking scripts
          WT-2921 Reduce the WT_SESSION hazard_size when possible
          WT-2923 heap-use-after-free on address in compaction
          WT-2924 Ensure we are doing eviction when threads are waiting for it
          WT-2925 WT_THREAD_PANIC_FAIL is a WT_THREAD structure flag
          WT-2926 WT_CONNECTION.reconfigure can attempt unlock of not-locked lock
          WT-2928 Eviction failing to switch queues can lead to starvation
          Branch: master
          https://github.com/mongodb/mongo/commit/9dda827a3ae58beef36d53da1b55554cbd8744c4

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'} Message: Import wiredtiger: fc0e7abe82595e579573d42448632f7b36a2d154 from branch mongodb-3.4 ref: 5bc03723a7..fc0e7abe82 for: 3.3.15 WT-2864 Reconfiguring the checkpoint server can lead to hangs WT-2874 Change test_compact01 to avoid eviction WT-2918 The dist scripts create C files s_whitespace complains about WT-2919 Don't mask error returns from style checking scripts WT-2921 Reduce the WT_SESSION hazard_size when possible WT-2923 heap-use-after-free on address in compaction WT-2924 Ensure we are doing eviction when threads are waiting for it WT-2925 WT_THREAD_PANIC_FAIL is a WT_THREAD structure flag WT-2926 WT_CONNECTION.reconfigure can attempt unlock of not-locked lock WT-2928 Eviction failing to switch queues can lead to starvation Branch: master https://github.com/mongodb/mongo/commit/9dda827a3ae58beef36d53da1b55554cbd8744c4

            People

            • Assignee:
              keith.bostic Keith Bostic
              Reporter:
              keith.bostic Keith Bostic
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: