Details

    • Type: Task
    • Status: Closed
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: WT2.5.2
    • Labels:

      Description

      Sanitizer failed with a memory leak of ikeys:

      http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/2733/

      We allocate them in __wt_row_ikey and there are a bunch of places we free them. We somehow missed 3 apparently.

      Here's the stack:

      Direct leak of 142 byte(s) in 3 object(s) allocated from:
          #0 0x48df19 in calloc (<http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/test/format/t+0x48df19)>
          WT-1 0x67d388 in __wt_calloc <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/../src/os_posix/os_alloc.c>:39
          WT-2 0xa70628 in __wt_row_ikey <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/../src/btree/row_key.c>:494
          WT-3 0x9e9d0d in __split_ref_deepen_move <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/../src/btree/bt_split.c>:294
          WT-4 0x9e70de in __split_deepen <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/../src/btree/bt_split.c>:503
          WT-5 0x9da638 in __split_parent <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/../src/btree/bt_split.c>:1052
          WT-6 0x9de764 in __wt_split_multi <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/../src/btree/bt_split.c>:1503
          WT-7 0x5c94da in __evict_page_dirty_update <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/../src/evict/evict_page.c>:215
          WT-8 0x5c2f85 in __wt_evict <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/../src/evict/evict_page.c>:109
          WT-9 0x5a7aba in __wt_evict_page <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/../src/evict/evict_lru.c>:630
          WT-10 0x5ab00b in __wt_evict_lru_page <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/../src/evict/evict_lru.c>:1310
          WT-11 0xd4df90 in __wt_cache_full_check <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/../src/include/cache.i>:193
          WT-12 0xd4fdce in __cursor_enter <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/../src/include/cursor.i>:63:33
          WT-13 0xd4efa6 in __curfile_enter <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/../src/include/cursor.i>:98:32
          WT-14 0xd2feee in __cursor_func_init <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/../src/include/cursor.i>:197:33
          WT-15 0xd2d7bc in __wt_btcur_search <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/../src/btree/bt_cursor.c>:308:32
          WT-16 0xb35693 in __curfile_search <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/../src/cursor/cur_file.c>:200
          WT-17 0x4bfdc2 in read_row <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/test/format/../../../test/format/ops.c>:602
          WT-18 0x4bd790 in ops <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/test/format/../../../test/format/ops.c>:476
          WT-19 0x3f17e07ee4 in start_thread (/lib64/libpthread.so.0+0x3f17e07ee4)
      

      Here's the CONFIG:

      ############################################
      #  RUN PARAMETERS
      ############################################
      abort=0
      auto_throttle=1
      firstfit=0
      bitcnt=2
      bloom=1
      bloom_bit_count=41
      bloom_hash_count=16
      bloom_oldest=0
      cache=13
      checkpoints=1
      checksum=uncompressed
      chunk_size=2
      compaction=0
      compression=zlib
      data_extend=0
      data_source=table
      delete_pct=12
      dictionary=0
      evict_max=3
      file_type=row-store
      backups=0
      huffman_key=0
      huffman_value=0
      insert_pct=14
      internal_key_truncation=1
      internal_page_max=16
      isolation=snapshot
      key_gap=18
      key_max=80
      key_min=26
      leak_memory=0
      leaf_page_max=9
      logging=0
      logging_archive=0
      logging_prealloc=0
      logging=0
      lsm_worker_threads=3
      merge_max=15
      mmap=1
      ops=100000
      prefix_compression=1
      prefix_compression_min=8
      repeat_data_pct=46
      reverse=0
      rows=100000
      runs=1
      split_pct=47
      statistics=0
      threads=25
      timer=0
      value_max=2874
      value_min=3
      wiredtiger_config=
      write_pct=20
      ############################################
      

        Issue Links

          Activity

          Hide
          sueloverso Sue Loverso added a comment -

          This is still happening even with all the recent split changes:
          http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/2842

          =================================================================
          ==20823==ERROR: LeakSanitizer: detected memory leaks
           
          Direct leak of 90 byte(s) in 2 object(s) allocated from:
              #0 0x484659 in calloc (/fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/t+0x484659)
              WT-1 0x678638 in __wt_calloc /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/os_posix/os_alloc.c:39
              WT-2 0xa72838 in __wt_row_ikey /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/row_key.c:494
              WT-3 0x9e9f27 in __split_ref_deepen_move /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_split.c:284
              WT-4 0x9e7290 in __split_deepen /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_split.c:494
              WT-5 0x9d9897 in __split_parent /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_split.c:1034
              WT-6 0x9dd765 in __wt_split_multi /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_split.c:1486
              WT-7 0x5c12ea in __evict_page_dirty_update /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_page.c:215
              WT-8 0x5bac1e in __wt_evict /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_page.c:109
              WT-9 0x59c73a in __wt_evict_page /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_lru.c:629
              WT-10 0x5a009f in __wt_evict_lru_page /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_lru.c:1313
              WT-11 0x5a2c25 in __wt_cache_wait /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_lru.c:1372
              WT-12 0xd4c902 in __wt_cache_full_check /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/include/cache.i:176
              WT-13 0xd4daae in __cursor_enter /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/include/cursor.i:63
              WT-14 0xd4cc86 in __curfile_enter /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/include/cursor.i:96
              WT-15 0xd2e6fe in __cursor_func_init /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/include/cursor.i:198
              WT-16 0xd3c2a4 in __wt_btcur_update /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_cursor.c:700
              WT-17 0xb40457 in __curfile_update /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/cursor/cur_file.c:284
              WT-18 0x4bda13 in row_update /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/ops.c:778
              WT-19 0x4b3844 in ops /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/ops.c:442
              WT-20 0x7f99d2212181 in start_thread /build/buildd/eglibc-2.19/nptl/pthread_create.c:312
          

          Here's is this CONFIG:

          ############################################
          #  RUN PARAMETERS
          ############################################
          abort=0
          auto_throttle=1
          firstfit=0
          bitcnt=6
          bloom=1
          bloom_bit_count=41
          bloom_hash_count=31
          bloom_oldest=0
          cache=16
          checkpoints=0
          checksum=uncompressed
          chunk_size=4
          compaction=0
          compression=zlib
          data_extend=0
          data_source=table
          delete_pct=10
          dictionary=0
          evict_max=4
          file_type=row-store
          backups=0
          huffman_key=0
          huffman_value=0
          insert_pct=32
          internal_key_truncation=1
          internal_page_max=17
          isolation=snapshot
          key_gap=8
          key_max=94
          key_min=26
          leak_memory=0
          leaf_page_max=9
          logging=1
          logging_archive=0
          logging_prealloc=0
          logging=1
          lsm_worker_threads=4
          merge_max=20
          mmap=1
          ops=100000
          prefix_compression=0
          prefix_compression_min=5
          repeat_data_pct=2
          reverse=0
          rows=100000
          runs=1
          split_pct=57
          statistics=0
          threads=28
          timer=0
          value_max=2604
          value_min=13
          wiredtiger_config=
          write_pct=50
          ############################################
          

          Show
          sueloverso Sue Loverso added a comment - This is still happening even with all the recent split changes: http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/2842 ================================================================= ==20823==ERROR: LeakSanitizer: detected memory leaks   Direct leak of 90 byte(s) in 2 object(s) allocated from: #0 0x484659 in calloc (/fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/t+0x484659) WT-1 0x678638 in __wt_calloc /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/os_posix/os_alloc.c:39 WT-2 0xa72838 in __wt_row_ikey /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/row_key.c:494 WT-3 0x9e9f27 in __split_ref_deepen_move /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_split.c:284 WT-4 0x9e7290 in __split_deepen /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_split.c:494 WT-5 0x9d9897 in __split_parent /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_split.c:1034 WT-6 0x9dd765 in __wt_split_multi /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_split.c:1486 WT-7 0x5c12ea in __evict_page_dirty_update /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_page.c:215 WT-8 0x5bac1e in __wt_evict /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_page.c:109 WT-9 0x59c73a in __wt_evict_page /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_lru.c:629 WT-10 0x5a009f in __wt_evict_lru_page /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_lru.c:1313 WT-11 0x5a2c25 in __wt_cache_wait /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_lru.c:1372 WT-12 0xd4c902 in __wt_cache_full_check /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/include/cache.i:176 WT-13 0xd4daae in __cursor_enter /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/include/cursor.i:63 WT-14 0xd4cc86 in __curfile_enter /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/include/cursor.i:96 WT-15 0xd2e6fe in __cursor_func_init /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/include/cursor.i:198 WT-16 0xd3c2a4 in __wt_btcur_update /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_cursor.c:700 WT-17 0xb40457 in __curfile_update /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/cursor/cur_file.c:284 WT-18 0x4bda13 in row_update /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/ops.c:778 WT-19 0x4b3844 in ops /fast/jenkins/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/ops.c:442 WT-20 0x7f99d2212181 in start_thread /build/buildd/eglibc-2.19/nptl/pthread_create.c:312 Here's is this CONFIG: ############################################ # RUN PARAMETERS ############################################ abort=0 auto_throttle=1 firstfit=0 bitcnt=6 bloom=1 bloom_bit_count=41 bloom_hash_count=31 bloom_oldest=0 cache=16 checkpoints=0 checksum=uncompressed chunk_size=4 compaction=0 compression=zlib data_extend=0 data_source=table delete_pct=10 dictionary=0 evict_max=4 file_type=row-store backups=0 huffman_key=0 huffman_value=0 insert_pct=32 internal_key_truncation=1 internal_page_max=17 isolation=snapshot key_gap=8 key_max=94 key_min=26 leak_memory=0 leaf_page_max=9 logging=1 logging_archive=0 logging_prealloc=0 logging=1 lsm_worker_threads=4 merge_max=20 mmap=1 ops=100000 prefix_compression=0 prefix_compression_min=5 repeat_data_pct=2 reverse=0 rows=100000 runs=1 split_pct=57 statistics=0 threads=28 timer=0 value_max=2604 value_min=13 wiredtiger_config= write_pct=50 ############################################
          Hide
          keith.bostic Keith Bostic added a comment -

          I added barrier instructions around the allocation in the WT_REF, in case there was some race I was missing:

          diff --git a/src/btree/bt_split.c b/src/btree/bt_split.c
          index 6ebd460..a947784 100644
          --- a/src/btree/bt_split.c
          +++ b/src/btree/bt_split.c
          @@ -260,7 +260,7 @@ __split_ref_deepen_move(WT_SESSION_IMPL *session,
                  WT_DECL_RET;
                  WT_IKEY *ikey;
                  size_t size;
          -       void *key;
          +       void *copy, *key;
           
                  /*
                   * Instantiate row-store keys, and column- and row-store addresses in
          @@ -279,10 +279,13 @@ __split_ref_deepen_move(WT_SESSION_IMPL *session,
                   * Row-store keys, first.
                   */
                  if (parent->type == WT_PAGE_ROW_INT) {
          +               copy = ref->key.ikey;
          +               WT_BARRIER();
                          if ((ikey = __wt_ref_key_instantiated(ref)) == NULL) {
                                  __wt_ref_key(parent, ref, &key, &size);
                                  WT_RET(__wt_row_ikey(session, 0, key, size, &ikey));
          -                       ref->key.ikey = ikey;
          +                       WT_ASSERT(session,
          +                           WT_ATOMIC_CAS8(ref->key.ikey, copy, ikey) != 0);
                          } else {
                                  WT_RET(__split_ovfl_key_cleanup(session, parent, ref));
                                  *parent_decrp += sizeof(WT_IKEY) + ikey->size;
          

          but that didn't help – we must be leaking a WT_REF somewhere.

          Show
          keith.bostic Keith Bostic added a comment - I added barrier instructions around the allocation in the WT_REF, in case there was some race I was missing: diff --git a/src/btree/bt_split.c b/src/btree/bt_split.c index 6ebd460..a947784 100644 --- a/src/btree/bt_split.c +++ b/src/btree/bt_split.c @@ -260,7 +260,7 @@ __split_ref_deepen_move(WT_SESSION_IMPL *session, WT_DECL_RET; WT_IKEY *ikey; size_t size; - void *key; + void *copy, *key; /* * Instantiate row-store keys, and column- and row-store addresses in @@ -279,10 +279,13 @@ __split_ref_deepen_move(WT_SESSION_IMPL *session, * Row-store keys, first. */ if (parent->type == WT_PAGE_ROW_INT) { + copy = ref->key.ikey; + WT_BARRIER(); if ((ikey = __wt_ref_key_instantiated(ref)) == NULL) { __wt_ref_key(parent, ref, &key, &size); WT_RET(__wt_row_ikey(session, 0, key, size, &ikey)); - ref->key.ikey = ikey; + WT_ASSERT(session, + WT_ATOMIC_CAS8(ref->key.ikey, copy, ikey) != 0); } else { WT_RET(__split_ovfl_key_cleanup(session, parent, ref)); *parent_decrp += sizeof(WT_IKEY) + ikey->size; but that didn't help – we must be leaking a WT_REF somewhere.
          Hide
          sueloverso Sue Loverso added a comment -

          [~michaelcahill] can this issue be closed?

          Show
          sueloverso Sue Loverso added a comment - [~michaelcahill] can this issue be closed?
          Hide
          michael.cahill Michael Cahill added a comment -

          Yes, it hasn't been seen since WT-1747 was merged, I believe it's fixed.

          Show
          michael.cahill Michael Cahill added a comment - Yes, it hasn't been seen since WT-1747 was merged, I believe it's fixed.
          Hide
          ramon.fernandez Ramon Fernandez added a comment -
          Show
          ramon.fernandez Ramon Fernandez added a comment - Additional ticket information from GitHub This ticket was referenced in the following commits: a326c3ba10e0d299944a650b890f8c2d851db34a da0bc67c821282e9fd0da725279811b59b25a675 fdead69550f9202541f4ea9dce396aa0003ed15a 16119f98fee9525bf1990ccbcc979b9f08a8b000

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since reply:
                1 year, 48 weeks, 6 days ago
                Date of 1st Reply: