Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-7392

Added evicted flag to dhandle for use by session sweep

    • 8

      Address sanitiser has detected a heap use after free:

      (gdb) bt 
      #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
      #1  0x00007f318dbd5921 in __GI_abort () at abort.c:79
      #2  0x0000558d5865c397 in __sanitizer::Abort() () at /data/mci/2b29d50424b17e9bd64c23288b01e972/toolchain-builder/tmp/build-llvm.sh-h2X/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cc:157
      #3  0x0000558d5865ade1 in __sanitizer::Die() () at /data/mci/2b29d50424b17e9bd64c23288b01e972/toolchain-builder/tmp/build-llvm.sh-h2X/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_termination.cc:59
      #4  0x0000558d58642dc9 in ~ScopedInErrorReport () at /data/mci/2b29d50424b17e9bd64c23288b01e972/toolchain-builder/tmp/build-llvm.sh-h2X/llvm/projects/compiler-rt/lib/asan/asan_report.cc:187
      #5  0x0000558d586445b3 in ReportGenericError () at /data/mci/2b29d50424b17e9bd64c23288b01e972/toolchain-builder/tmp/build-llvm.sh-h2X/llvm/projects/compiler-rt/lib/asan/asan_report.cc:464
      #6  0x0000558d58644ebb in __asan_report_load8 () at /data/mci/2b29d50424b17e9bd64c23288b01e972/toolchain-builder/tmp/build-llvm.sh-h2X/llvm/projects/compiler-rt/lib/asan/asan_rtl.cc:121
      #7  0x0000558d5b40d534 in __wt_btree_bytes_evictable (session=<optimized out>) at src/third_party/wiredtiger/src/include/btree_inline.h:142
      #8  __session_dhandle_sweep (session=<optimized out>) at src/third_party/wiredtiger/src/session/session_dhandle.c:384
      #9  __session_get_dhandle (session=<optimized out>, uri=<optimized out>, checkpoint=<optimized out>) at src/third_party/wiredtiger/src/session/session_dhandle.c:439
      #10 __wt_session_get_dhandle (session=0x7f317ed31050, uri=0x607000575b60 "table:collection-1023--1887297523114510167", checkpoint=0x0, cfg=0x0, flags=0) at src/third_party/wiredtiger/src/session/session_dhandle.c:474
      #11 0x0000558d5b396050 in __wt_schema_get_table_uri (session=0x7f317ed31050, uri=0x7f314d00f8f0 "\003B\200", ok_incomplete=false, flags=0, tablep=0x7f314d010c30) at src/third_party/wiredtiger/src/schema/schema_list.c:27
      #12 0x0000558d5b1fbfcd in __wt_curtable_open (session=0x7f317ed31050, uri=0x607000575b60 "table:collection-1023--1887297523114510167", owner=0x0, cfg=0x7f314d010fa0, cursorp=0x7f314d010f80) at src/third_party/wiredtiger/src/cursor/cur_table.c:990
      #13 0x0000558d5b3b6177 in __session_open_cursor_int (session=0x7f317ed31050, uri=0x607000575b60 "table:collection-1023--1887297523114510167", owner=0x0, other=0x0, cfg=0x7f314d010fa0, cursorp=0x7f314d010f80)
          at src/third_party/wiredtiger/src/session/session_api.c:453
      #14 0x0000558d5b3b8783 in __session_open_cursor (wt_session=0x7f317ed31050, uri=0x607000575b60 "table:collection-1023--1887297523114510167", to_dup=0x0, config=0x7f3100000000 <error: Cannot access memory at address 0x7f3100000000>, 
          cursorp=0x7f314d011320) at src/third_party/wiredtiger/src/session/session_api.c:603
      

      Looking at the coredump at frame 8, the heap use after free shows that a dhandle is pointed at undefined memory:

      (gdb) list
      379	    TAILQ_FOREACH_SAFE(dhandle_cache, &session->dhandles, q, dhandle_cache_tmp)
      380	    {
      381	        dhandle = dhandle_cache->dhandle;
      382	        empty_btree = false;
      383	        if (dhandle->type == WT_DHANDLE_TYPE_BTREE)
      384	            WT_WITH_DHANDLE(
      385	              session, dhandle, empty_btree = (__wt_btree_bytes_evictable(session) == 0));
      386	
      387	        if (dhandle != session->dhandle && dhandle->session_inuse == 0 &&
      388	          (WT_DHANDLE_INACTIVE(dhandle) ||
      (gdb) print *dhandle      
      Cannot access memory at address 0x8
      (gdb) f 7
      #7  0x0000558d5b40d534 in __wt_btree_bytes_evictable (session=<optimized out>) at src/third_party/wiredtiger/src/include/btree_inline.h:142
      142	    bytes_root = root_page == NULL ? 0 : root_page->memory_footprint;
      (gdb) print btree->dhandle->name
      $20 = 0x60400142d510 "file:index-1063--1887297523114510167.wt"
      

      The address sanitiser has information about how the dhandle is created and freed.
      Freeing stacktrace:

      [j0:prim] 0x6070004c7a68 is located 40 bytes inside of 80-byte region [0x6070004c7a40,0x6070004c7a90)
      [j0:prim] freed by thread T13 here:
      [j0:prim]     #0 0x558d5863e462 in free /data/mci/2b29d50424b17e9bd64c23288b01e972/toolchain-builder/tmp/build-llvm.sh-h2X/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:124:3
      [j0:prim]     #1 0x558d5afc8a32 in __wt_page_out /data/mci/13e69e2ed46658a72d1a2c0c7c95251e/src/src/third_party/wiredtiger/src/btree/bt_discard.c:133:5
      [j0:prim]     #2 0x558d5b2153eb in __wt_evict_file /data/mci/13e69e2ed46658a72d1a2c0c7c95251e/src/src/third_party/wiredtiger/src/evict/evict_file.c:106:13
      [j0:prim]     #3 0x558d5b0de604 in __wt_conn_dhandle_close /data/mci/13e69e2ed46658a72d1a2c0c7c95251e/src/src/third_party/wiredtiger/src/conn/conn_dhandle.c:401:9
      [j0:prim]     #4 0x558d5b0ff1ff in __sweep_discard_trees /data/mci/13e69e2ed46658a72d1a2c0c7c95251e/src/src/third_party/wiredtiger/src/conn/conn_sweep.c:161:9
      [j0:prim]     #5 0x558d5b0ff1ff in __sweep_server /data/mci/13e69e2ed46658a72d1a2c0c7c95251e/src/src/third_party/wiredtiger/src/conn/conn_sweep.c:323
      [j0:prim]     #6 0x7f318d7756da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
      

      Allocating stacktrace:

      [j0:prim] previously allocated by thread T1141 (conn1090) here:
      [j0:prim]     #0 0x558d5863e9da in calloc /data/mci/2b29d50424b17e9bd64c23288b01e972/toolchain-builder/tmp/build-llvm.sh-h2X/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:155:3
      [j0:prim]     #1 0x558d5b2f5620 in __wt_calloc /data/mci/13e69e2ed46658a72d1a2c0c7c95251e/src/src/third_party/wiredtiger/src/os_common/os_alloc.c:50:14
      [j0:prim]     #2 0x558d5afe3451 in __wt_page_alloc /data/mci/13e69e2ed46658a72d1a2c0c7c95251e/src/src/third_party/wiredtiger/src/btree/bt_page.c:63:5
      [j0:prim]     #3 0x558d5afe49c0 in __wt_page_inmem /data/mci/13e69e2ed46658a72d1a2c0c7c95251e/src/src/third_party/wiredtiger/src/btree/bt_page.c:187:5
      [j0:prim]     #4 0x558d5afd64d9 in __wt_btree_tree_open /data/mci/13e69e2ed46658a72d1a2c0c7c95251e/src/src/third_party/wiredtiger/src/btree/bt_handle.c:697:5
      [j0:prim]     #5 0x558d5afcb5df in __wt_btree_open /data/mci/13e69e2ed46658a72d1a2c0c7c95251e/src/src/third_party/wiredtiger/src/btree/bt_handle.c:150:13
      [j0:prim]     #6 0x558d5b0df990 in __wt_conn_dhandle_open /data/mci/13e69e2ed46658a72d1a2c0c7c95251e/src/src/third_party/wiredtiger/src/conn/conn_dhandle.c:531:9
      [j0:prim]     #7 0x558d5b40c6d2 in __wt_session_get_dhandle /data/mci/13e69e2ed46658a72d1a2c0c7c95251e/src/src/third_party/wiredtiger/src/session/session_dhandle.c:510:20
      [j0:prim]     #8 0x558d5b40d24c in __wt_session_get_dhandle /data/mci/13e69e2ed46658a72d1a2c0c7c95251e/src/src/third_party/wiredtiger/src/session/session_dhandle.c:503:13
      [j0:prim]     #9 0x558d5b40b383 in __wt_session_get_btree_ckpt /data/mci/13e69e2ed46658a72d1a2c0c7c95251e/src/src/third_party/wiredtiger/src/session/session_dhandle.c:319:11
      [j0:prim]     #10 0x558d5b144a2b in __wt_curfile_open /data/mci/13e69e2ed46658a72d1a2c0c7c95251e/src/src/third_party/wiredtiger/src/include/time_inline.h
      [j0:prim]     #11 0x558d5b3b5c39 in __session_open_cursor_int /data/mci/13e69e2ed46658a72d1a2c0c7c95251e/src/src/third_party/wiredtiger/src/session/session_api.c:487:13
      [j0:prim]     #12 0x558d5b1fc12f in __wt_curtable_open /data/mci/13e69e2ed46658a72d1a2c0c7c95251e/src/src/third_party/wiredtiger/src/cursor/cur_table.c:1000:15
      [j0:prim]     #13 0x558d5b3b6176 in __session_open_cursor_int /data/mci/13e69e2ed46658a72d1a2c0c7c95251e/src/src/third_party/wiredtiger/src/session/session_api.c:453:13
      [j0:prim]     #14 0x558d5b3b8782 in __session_open_cursor /data/mci/13e69e2ed46658a72d1a2c0c7c95251e/src/src/third_party/wiredtiger/src/session/session_api.c:603:5
      

      The sweep server is looking for dead or excess dhandles to discard, and decides to free this particular dhandle. Furthermore, the stacktrace performs a cursor open on a dhandle that we have previously discarded, producing an invalid state, where wiredtiger is trying to open up a cursor on a dhandle that has been freed already.

            Assignee:
            donald.anderson@mongodb.com Donald Anderson
            Reporter:
            jie.chen@mongodb.com Jie Chen
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: