Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: WT2.7.0
    • Labels:
      None
    • # Replies:
      10
    • Last comment by Customer:
      true

      Description

      It is possible to generate a memory leak in WiredTiger with wtperf.

      The leaks are:

      ==9930==ERROR: LeakSanitizer: detected memory leaks
       
      Direct leak of 111024 byte(s) in 6939 object(s) allocated from:
          #0 0x4a5880 in calloc (/home/alexg/work/wiredtiger/build_posix/bench/wtperf/wtperf+0x4a5880)
          #1 0x707c0f in __wt_calloc /home/alexg/work/wiredtiger/build_posix/../src/os_posix/os_alloc.c:60:11
          #2 0xacfd7a in __wt_multi_to_ref /home/alexg/work/wiredtiger/build_posix/../src/btree/bt_split.c:1574:3
          #3 0xadef61 in __split_multi /home/alexg/work/wiredtiger/build_posix/../src/btree/bt_split.c:1929:3
          #4 0xadd7e4 in __wt_split_multi /home/alexg/work/wiredtiger/build_posix/../src/btree/bt_split.c:1980:13
          #5 0x64515a in __evict_page_dirty_update /home/alexg/work/wiredtiger/build_posix/../src/evict/evict_page.c:296:4
          #6 0x63ea9e in __wt_evict /home/alexg/work/wiredtiger/build_posix/../src/evict/evict_page.c:123:3
          #7 0x9eb58b in __wt_delete_page /home/alexg/work/wiredtiger/build_posix/../src/btree/bt_delete.c:80:9
          #8 0xb6a1ec in __wt_tree_walk /home/alexg/work/wiredtiger/build_posix/../src/btree/bt_walk.c:270:5
          #9 0xe71a87 in __wt_btcur_next /home/alexg/work/wiredtiger/build_posix/../src/btree/bt_curnext.c:530:3
          #10 0xee1f0f in __cursor_truncate /home/alexg/work/wiredtiger/build_posix/../src/btree/bt_cursor.c:1009:14
          #11 0xee00d9 in __wt_btcur_range_truncate /home/alexg/work/wiredtiger/build_posix/../src/btree/bt_cursor.c:1138:3
          #12 0xe01a93 in __wt_schema_range_truncate /home/alexg/work/wiredtiger/build_posix/../src/schema/schema_truncate.c:181:3
          #13 0x8a3394 in __session_truncate /home/alexg/work/wiredtiger/build_posix/../src/session/session_api.c:910:2
          #14 0x510d7d in run_truncate /home/alexg/work/wiredtiger/build_posix/bench/wtperf/../../../bench/wtperf/wtperf_truncate.c:187:13
          #15 0x4fd892 in worker /home/alexg/work/wiredtiger/build_posix/bench/wtperf/../../../bench/wtperf/wtperf.c:556:15
          #16 0x7fd6cffc1554 in start_thread (/lib64/libpthread.so.0+0x7554)
       
      Direct leak of 17712 byte(s) in 1107 object(s) allocated from:
          #0 0x4a5880 in calloc (/home/alexg/work/wiredtiger/build_posix/bench/wtperf/wtperf+0x4a5880)
          #1 0x707c0f in __wt_calloc /home/alexg/work/wiredtiger/build_posix/../src/os_posix/os_alloc.c:60:11
          #2 0xacfd7a in __wt_multi_to_ref /home/alexg/work/wiredtiger/build_posix/../src/btree/bt_split.c:1574:3
          #3 0xadef61 in __split_multi /home/alexg/work/wiredtiger/build_posix/../src/btree/bt_split.c:1929:3
          #4 0xadd7e4 in __wt_split_multi /home/alexg/work/wiredtiger/build_posix/../src/btree/bt_split.c:1980:13
          #5 0x64515a in __evict_page_dirty_update /home/alexg/work/wiredtiger/build_posix/../src/evict/evict_page.c:296:4
          #6 0x63ea9e in __wt_evict /home/alexg/work/wiredtiger/build_posix/../src/evict/evict_page.c:123:3
          #7 0xeeaa39 in __wt_page_release_evict /home/alexg/work/wiredtiger/build_posix/../src/include/btree.i:1141:13
          #8 0xee8843 in __wt_page_release /home/alexg/work/wiredtiger/build_posix/../src/include/btree.i:1203:2
          #9 0xed36bc in __curfile_leave /home/alexg/work/wiredtiger/build_posix/../src/include/cursor.i:133:8
          #10 0xecb736 in __cursor_func_init /home/alexg/work/wiredtiger/build_posix/../src/include/cursor.i:250:3
          #11 0xed64ed in __wt_btcur_remove /home/alexg/work/wiredtiger/build_posix/../src/btree/bt_cursor.c:657:8
          #12 0xee1c5f in __cursor_truncate /home/alexg/work/wiredtiger/build_posix/../src/btree/bt_cursor.c:1000:8
          #13 0xee00d9 in __wt_btcur_range_truncate /home/alexg/work/wiredtiger/build_posix/../src/btree/bt_cursor.c:1138:3
          #14 0xe01a93 in __wt_schema_range_truncate /home/alexg/work/wiredtiger/build_posix/../src/schema/schema_truncate.c:181:3
          #15 0x8a3394 in __session_truncate /home/alexg/work/wiredtiger/build_posix/../src/session/session_api.c:910:2
          #16 0x510d7d in run_truncate /home/alexg/work/wiredtiger/build_posix/bench/wtperf/../../../bench/wtperf/wtperf_truncate.c:187:13
          #17 0x4fd892 in worker /home/alexg/work/wiredtiger/build_posix/bench/wtperf/../../../bench/wtperf/wtperf.c:556:15
          #18 0x7fd6cffc1554 in start_thread (/lib64/libpthread.so.0+0x7554)
       
      Indirect leak of 69362 byte(s) in 6939 object(s) allocated from:
          #0 0x4a5880 in calloc (/home/alexg/work/wiredtiger/build_posix/bench/wtperf/wtperf+0x4a5880)
          #1 0x707c0f in __wt_calloc /home/alexg/work/wiredtiger/build_posix/../src/os_posix/os_alloc.c:60:11
          #2 0x70b32f in __wt_strndup /home/alexg/work/wiredtiger/build_posix/../src/os_posix/os_alloc.c:224:2
          #3 0xad0302 in __wt_multi_to_ref /home/alexg/work/wiredtiger/build_posix/../src/btree/bt_split.c:1578:3
          #4 0xadef61 in __split_multi /home/alexg/work/wiredtiger/build_posix/../src/btree/bt_split.c:1929:3
          #5 0xadd7e4 in __wt_split_multi /home/alexg/work/wiredtiger/build_posix/../src/btree/bt_split.c:1980:13
          #6 0x64515a in __evict_page_dirty_update /home/alexg/work/wiredtiger/build_posix/../src/evict/evict_page.c:296:4
          #7 0x63ea9e in __wt_evict /home/alexg/work/wiredtiger/build_posix/../src/evict/evict_page.c:123:3
          #8 0x9eb58b in __wt_delete_page /home/alexg/work/wiredtiger/build_posix/../src/btree/bt_delete.c:80:9
          #9 0xb6a1ec in __wt_tree_walk /home/alexg/work/wiredtiger/build_posix/../src/btree/bt_walk.c:270:5
          #10 0xe71a87 in __wt_btcur_next /home/alexg/work/wiredtiger/build_posix/../src/btree/bt_curnext.c:530:3
          #11 0xee1f0f in __cursor_truncate /home/alexg/work/wiredtiger/build_posix/../src/btree/bt_cursor.c:1009:14
          #12 0xee00d9 in __wt_btcur_range_truncate /home/alexg/work/wiredtiger/build_posix/../src/btree/bt_cursor.c:1138:3
          #13 0xe01a93 in __wt_schema_range_truncate /home/alexg/work/wiredtiger/build_posix/../src/schema/schema_truncate.c:181:3
          #14 0x8a3394 in __session_truncate /home/alexg/work/wiredtiger/build_posix/../src/session/session_api.c:910:2
          #15 0x510d7d in run_truncate /home/alexg/work/wiredtiger/build_posix/bench/wtperf/../../../bench/wtperf/wtperf_truncate.c:187:13
          #16 0x4fd892 in worker /home/alexg/work/wiredtiger/build_posix/bench/wtperf/../../../bench/wtperf/wtperf.c:556:15
          #17 0x7fd6cffc1554 in start_thread (/lib64/libpthread.so.0+0x7554)
       
      Indirect leak of 10524 byte(s) in 1107 object(s) allocated from:
          #0 0x4a5880 in calloc (/home/alexg/work/wiredtiger/build_posix/bench/wtperf/wtperf+0x4a5880)
          #1 0x707c0f in __wt_calloc /home/alexg/work/wiredtiger/build_posix/../src/os_posix/os_alloc.c:60:11
          #2 0x70b32f in __wt_strndup /home/alexg/work/wiredtiger/build_posix/../src/os_posix/os_alloc.c:224:2
          #3 0xad0302 in __wt_multi_to_ref /home/alexg/work/wiredtiger/build_posix/../src/btree/bt_split.c:1578:3
          #4 0xadef61 in __split_multi /home/alexg/work/wiredtiger/build_posix/../src/btree/bt_split.c:1929:3
          #5 0xadd7e4 in __wt_split_multi /home/alexg/work/wiredtiger/build_posix/../src/btree/bt_split.c:1980:13
          #6 0x64515a in __evict_page_dirty_update /home/alexg/work/wiredtiger/build_posix/../src/evict/evict_page.c:296:4
          #7 0x63ea9e in __wt_evict /home/alexg/work/wiredtiger/build_posix/../src/evict/evict_page.c:123:3
          #8 0xeeaa39 in __wt_page_release_evict /home/alexg/work/wiredtiger/build_posix/../src/include/btree.i:1141:13
          #9 0xee8843 in __wt_page_release /home/alexg/work/wiredtiger/build_posix/../src/include/btree.i:1203:2
          #10 0xed36bc in __curfile_leave /home/alexg/work/wiredtiger/build_posix/../src/include/cursor.i:133:8
          #11 0xecb736 in __cursor_func_init /home/alexg/work/wiredtiger/build_posix/../src/include/cursor.i:250:3
          #12 0xed64ed in __wt_btcur_remove /home/alexg/work/wiredtiger/build_posix/../src/btree/bt_cursor.c:657:8
          #13 0xee1c5f in __cursor_truncate /home/alexg/work/wiredtiger/build_posix/../src/btree/bt_cursor.c:1000:8
          #14 0xee00d9 in __wt_btcur_range_truncate /home/alexg/work/wiredtiger/build_posix/../src/btree/bt_cursor.c:1138:3
          #15 0xe01a93 in __wt_schema_range_truncate /home/alexg/work/wiredtiger/build_posix/../src/schema/schema_truncate.c:181:3
          #16 0x8a3394 in __session_truncate /home/alexg/work/wiredtiger/build_posix/../src/session/session_api.c:910:2
          #17 0x510d7d in run_truncate /home/alexg/work/wiredtiger/build_posix/bench/wtperf/../../../bench/wtperf/wtperf_truncate.c:187:13
          #18 0x4fd892 in worker /home/alexg/work/wiredtiger/build_posix/bench/wtperf/../../../bench/wtperf/wtperf.c:556:15
          #19 0x7fd6cffc1554 in start_thread (/lib64/libpthread.so.0+0x7554)
       
      SUMMARY: AddressSanitizer: 208622 byte(s) leaked in 16092 allocation(s).
      

      The workload I'm using to reliably generate the memory leak is:

      conn_config="cache_size=1GB,checkpoint=(wait=5)"
      table_config="type=file"
      icount=500000
      report_interval=5
      run_time=20
      populate_threads=1
      # A high insert throughput, but not maxed out. So truncate can keep up.
      threads=((count=2,reads=1,ops_per_txn=100),(count=3,inserts=1,throttle=80000),(count=1,truncate=1,truncate_pct=10,truncate_count=1000000))
      

        Issue Links

          Activity

          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: Merge pull request #2350 from wiredtiger/WT-2251-ref-addr-leak
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/e731ef8ab8b8f9d1c65380c83fc3e7b318f3fbe8

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: Merge pull request #2350 from wiredtiger/ WT-2251 -ref-addr-leak Branch: develop https://github.com/wiredtiger/wiredtiger/commit/e731ef8ab8b8f9d1c65380c83fc3e7b318f3fbe8
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: Import wiredtiger-wiredtiger-mongodb-3.2-rc4-56-g7a4f325.tar.gz from wiredtiger branch mongodb-3.2

          ref: 8326df6..7a4f325

          e731ef8 WT-2251 Free addresses when discarding deleted page references.
          0e93d60 SERVER-21691 Avoid insert stalls
          264ec21 WT-2249 Keep eviction stuck until cache usage is under 100%.
          dca1411 WT-2250 Minor fix: use SET instead of INCRV for stat.
          Branch: master
          https://github.com/mongodb/mongo/commit/86e7b69a6c52c926d28a60d816faefa6db81eb96

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: Import wiredtiger-wiredtiger-mongodb-3.2-rc4-56-g7a4f325.tar.gz from wiredtiger branch mongodb-3.2 ref: 8326df6..7a4f325 e731ef8 WT-2251 Free addresses when discarding deleted page references. 0e93d60 SERVER-21691 Avoid insert stalls 264ec21 WT-2249 Keep eviction stuck until cache usage is under 100%. dca1411 WT-2250 Minor fix: use SET instead of INCRV for stat. Branch: master https://github.com/mongodb/mongo/commit/86e7b69a6c52c926d28a60d816faefa6db81eb96
          Hide
          milkie Eric Milkie added a comment -
          Show
          milkie Eric Milkie added a comment - rollback4.js is still running out of memory, even after this commit: https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_linux_64_amazon_ami_slow2_WT_86e7b69a6c52c926d28a60d816faefa6db81eb96_15_12_01_09_17_39
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: Merge pull request #2350 from wiredtiger/WT-2251-ref-addr-leak

          (cherry picked from commit e731ef8ab)

          Conflicts:
          src/btree/bt_discard.c
          src/btree/bt_slvg.c
          src/btree/bt_split.c
          src/evict/evict_page.c
          src/reconcile/rec_write.c
          Branch: mongodb-3.0
          https://github.com/wiredtiger/wiredtiger/commit/6feaa2812e1c02fe4b74bcd6c6baeda9f310899b

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: Merge pull request #2350 from wiredtiger/ WT-2251 -ref-addr-leak (cherry picked from commit e731ef8ab) Conflicts: src/btree/bt_discard.c src/btree/bt_slvg.c src/btree/bt_split.c src/evict/evict_page.c src/reconcile/rec_write.c Branch: mongodb-3.0 https://github.com/wiredtiger/wiredtiger/commit/6feaa2812e1c02fe4b74bcd6c6baeda9f310899b
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: Import wiredtiger-wiredtiger-mongodb-3.0.7-20-g27d0cbd.tar.gz from wiredtiger branch mongodb-3.0

          ref: deb2d81..27d0cbd

          6feaa28 WT-2251 Fix leaf of deleted page addresses.
          1da2d3a SERVER-21568 Fix a use-after-free.
          4228295 SERVER-21553 Free blocks during reverse splits.
          0398515 WT-2241 Use a lock to protect transaction ID allocation.
          06a5c7b WT-2237 Avoid yields if we race allocating transaction IDs.
          16a418b WT-2237 Have threads publish unique transaction IDs so that updates always become visible immediately on commit.
          Branch: v3.0
          https://github.com/mongodb/mongo/commit/eb8bc24a0d36b673022faa83976a4b33bc8f676b

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: Import wiredtiger-wiredtiger-mongodb-3.0.7-20-g27d0cbd.tar.gz from wiredtiger branch mongodb-3.0 ref: deb2d81..27d0cbd 6feaa28 WT-2251 Fix leaf of deleted page addresses. 1da2d3a SERVER-21568 Fix a use-after-free. 4228295 SERVER-21553 Free blocks during reverse splits. 0398515 WT-2241 Use a lock to protect transaction ID allocation. 06a5c7b WT-2237 Avoid yields if we race allocating transaction IDs. 16a418b WT-2237 Have threads publish unique transaction IDs so that updates always become visible immediately on commit. Branch: v3.0 https://github.com/mongodb/mongo/commit/eb8bc24a0d36b673022faa83976a4b33bc8f676b

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since reply:
                1 year, 16 weeks, 1 day ago
                Date of 1st Reply: