Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-1852

reconciliation using free'd memory

    • Type: Icon: Task Task
    • Resolution: Done
    • WT2.6.0
    • Affects Version/s: None
    • Component/s: None
    • Labels:

      From @markbenvenuto:

      For test/format, I am hitting an error because I believe of a bug with realloc invalidating pointers:

      There is a segfault here:

      Child-SP          RetAddr           Call Site
      00000034`fe07f108 00007ff6`5c340bb3 t_format!MoveSmall+0x2f8 [f:\dd\vctools\crt\crtw32\string\amd64\memcpy.asm @ 570]
      00000034`fe07f110 00007ff6`5c33fdd3 t_format!__rec_split_fixup+0x153 [d:\wt\wiredtiger\src\reconcile\rec_write.c @ 2678]
      00000034`fe07f1a0 00007ff6`5c33f367 t_format!__rec_split+0x393 [d:\wt\wiredtiger\src\reconcile\rec_write.c @ 2028]
      00000034`fe07f240 00007ff6`5c33e230 t_format!__rec_row_leaf_insert+0x2e7 [d:\wt\wiredtiger\src\reconcile\rec_write.c @ 4690]
      00000034`fe07f2d0 00007ff6`5c34740e t_format!__rec_row_leaf+0x210 [d:\wt\wiredtiger\src\reconcile\rec_write.c @ 4293]
      00000034`fe07f4b0 00007ff6`5c33755a t_format!__wt_reconcile+0x62e [d:\wt\wiredtiger\src\reconcile\rec_write.c @ 413]
      00000034`fe07f580 00007ff6`5c2ff459 t_format!__wt_evict_file+0x10a [d:\wt\wiredtiger\src\evict\evict_file.c @ 60]
      00000034`fe07f5f0 00007ff6`5c29d05c t_format!__wt_cache_op+0xc9 [d:\wt\wiredtiger\src\btree\bt_sync.c @ 261]
      00000034`fe07f640 00007ff6`5c29d983 t_format!__checkpoint_worker+0xdac [d:\wt\wiredtiger\src\txn\txn_ckpt.c @ 984]
      00000034`fe07f850 00007ff6`5c2a7c07 t_format!__wt_checkpoint_close+0x163 [d:\wt\wiredtiger\src\txn\txn_ckpt.c @ 1133]
      00000034`fe07f8b0 00007ff6`5c2a7e7c t_format!__wt_conn_btree_sync_and_close+0xe7 [d:\wt\wiredtiger\src\conn\conn_dhandle.c @ 276]
      00000034`fe07f900 00007ff6`5c2f6307 t_format!__wt_conn_dhandle_close_all+0x16c [d:\wt\wiredtiger\src\conn\conn_dhandle.c @ 672]
      00000034`fe07f970 00007ff6`5c2baf51 t_format!__wt_schema_worker+0x1c7 [d:\wt\wiredtiger\src\schema\schema_worker.c @ 54]
      00000034`fe07fa90 00007ff6`5c2f686e t_format!__wt_lsm_tree_worker+0x271 [d:\wt\wiredtiger\src\lsm\lsm_tree.c @ 1413]
      00000034`fe07fb30 00007ff6`5c2f6a83 t_format!__wt_schema_worker+0x72e [d:\wt\wiredtiger\src\schema\schema_worker.c @ 87]
      00000034`fe07fc50 00007ff6`5c28b998 t_format!__wt_schema_worker+0x943 [d:\wt\wiredtiger\src\schema\schema_worker.c @ 108]
      00000034`fe07fd70 00007ff6`5c26bdea t_format!__session_verify+0x2a8 [d:\wt\wiredtiger\src\session\session_api.c @ 746]
      00000034`fe07fe30 00007ff6`5c267d32 t_format!wts_verify+0xaa [d:\wt\wiredtiger\test\format\wts.c @ 465]
      00000034`fe07fe90 00007ff6`5c3626e4 t_format!main+0x4d2 [d:\wt\wiredtiger\test\format\t.c @ 217]
      00000034`fe07ff20 00007fff`549b16ad t_format!__tmainCRTStartup+0x144 [f:\dd\vctools\crt\crtw32\startup\crt0.c @ 255]
      00000034`fe07ff60 00007fff`54d6e954 KERNEL32!BaseThreadInitThunk+0xd
      00000034`fe07ff90 00000000`00000000 ntdll!RtlUserThreadStart+0x34
      

      for this code:

      	dsk_start = WT_PAGE_HEADER_BYTE(btree, dsk);
      	for (i = 0, bnd = r->bnd; i < r->bnd_next; ++i, ++bnd) {
      		/* Copy the page contents to the temporary buffer. */
      		len = WT_PTRDIFF32((bnd + 1)->start, bnd->start);
      		memcpy(dsk_start, bnd->start, len);
      

      where bnd->start points to freed memory.
      The memory was freed here:

      0:000> !heap -p -a 0x3496cc0628
          address 0000003496cc0628 found in
          _DPH_HEAP_ROOT @ 3480001000
          in free-ed allocation (  DPH_HEAP_BLOCK:         VirtAddr         VirtSize)
                                       3499c20f70:       3496cc0000             2000
          00007fff54df24dd ntdll!RtlDebugReAllocateHeap+0x0000000000000055
          00007fff54da73ae ntdll!RtlReAllocateHeap+0x000000000007c71e
          00007ff65c36391b t_format!realloc+0x0000000000000057 [f:\dd\vctools\crt\crtw32\heap\realloc.c @ 85]
          00007ff65c282c4d t_format!__wt_realloc+0x000000000000010d [d:\wt\wiredtiger\src\os_posix\os_alloc.c @ 81]
          00007ff65c282d26 t_format!__wt_realloc_aligned+0x0000000000000046 [d:\wt\wiredtiger\src\os_posix\os_alloc.c @ 182]
          00007ff65c28fcf2 t_format!__wt_buf_grow_worker+0x0000000000000102 [d:\wt\wiredtiger\src\support\scratch.c @ 46]
          00007ff65c2dd654 t_format!__wt_buf_grow+0x0000000000000094 [d:\wt\wiredtiger\src\include\buf.i @ 18]
          00007ff65c340e9c t_format!__rec_split_grow+0x00000000000000cc [d:\wt\wiredtiger\src\reconcile\rec_write.c @ 1916]
          00007ff65c34009b t_format!__rec_split+0x000000000000065b [d:\wt\wiredtiger\src\reconcile\rec_write.c @ 2099]
          00007ff65c33f367 t_format!__rec_row_leaf_insert+0x00000000000002e7 [d:\wt\wiredtiger\src\reconcile\rec_write.c @ 4690]
          00007ff65c33e230 t_format!__rec_row_leaf+0x0000000000000210 [d:\wt\wiredtiger\src\reconcile\rec_write.c @ 4293]
          00007ff65c34740e t_format!__wt_reconcile+0x000000000000062e [d:\wt\wiredtiger\src\reconcile\rec_write.c @ 413]
          00007ff65c33755a t_format!__wt_evict_file+0x000000000000010a [d:\wt\wiredtiger\src\evict\evict_file.c @ 60]
          00007ff65c2ff459 t_format!__wt_cache_op+0x00000000000000c9 [d:\wt\wiredtiger\src\btree\bt_sync.c @ 261]
          00007ff65c29d05c t_format!__checkpoint_worker+0x0000000000000dac [d:\wt\wiredtiger\src\txn\txn_ckpt.c @ 984]
          00007ff65c29d983 t_format!__wt_checkpoint_close+0x0000000000000163 [d:\wt\wiredtiger\src\txn\txn_ckpt.c @ 1133]
          00007ff65c2a7c07 t_format!__wt_conn_btree_sync_and_close+0x00000000000000e7 [d:\wt\wiredtiger\src\conn\conn_dhandle.c @ 276]
          00007ff65c2a7e7c t_format!__wt_conn_dhandle_close_all+0x000000000000016c [d:\wt\wiredtiger\src\conn\conn_dhandle.c @ 672]
          00007ff65c2f6307 t_format!__wt_schema_worker+0x00000000000001c7 [d:\wt\wiredtiger\src\schema\schema_worker.c @ 54]
          00007ff65c2baf51 t_format!__wt_lsm_tree_worker+0x0000000000000271 [d:\wt\wiredtiger\src\lsm\lsm_tree.c @ 1413]
          00007ff65c2f686e t_format!__wt_schema_worker+0x000000000000072e [d:\wt\wiredtiger\src\schema\schema_worker.c @ 87]
          00007ff65c2f6a83 t_format!__wt_schema_worker+0x0000000000000943 [d:\wt\wiredtiger\src\schema\schema_worker.c @ 108]
          00007ff65c28b998 t_format!__session_verify+0x00000000000002a8 [d:\wt\wiredtiger\src\session\session_api.c @ 746]
          00007ff65c26bdea t_format!wts_verify+0x00000000000000aa [d:\wt\wiredtiger\test\format\wts.c @ 465]
          00007ff65c267d32 t_format!main+0x00000000000004d2 [d:\wt\wiredtiger\test\format\t.c @ 217]
          00007ff65c3626e4 t_format!__tmainCRTStartup+0x0000000000000144 [f:\dd\vctools\crt\crtw32\startup\crt0.c @ 255]
          00007fff549b16ad KERNEL32!BaseThreadInitThunk+0x000000000000000d
          00007fff54d6e954 ntdll!RtlUserThreadStart+0x0000000000000034
      

      So i believe there an issue here:

      /*
       * __rec_split_grow --
       *	Grow the split buffer.
       */
      static int
      __rec_split_grow(WT_SESSION_IMPL *session, WT_RECONCILE *r, size_t add_len)
      {
      	WT_BM *bm;
      	WT_BTREE *btree;
      	size_t corrected_page_size, len;
      
      	btree = S2BT(session);
      	bm = btree->bm;
      
      	len = WT_PTRDIFF(r->first_free, r->dsk.mem);
      	corrected_page_size = len + add_len;
      	WT_RET(bm->write_size(bm, session, &corrected_page_size));
      >>>	WT_RET(__wt_buf_grow(session, &r->dsk, corrected_page_size));
      	r->first_free = (uint8_t *)r->dsk.mem + len;
      	WT_ASSERT(session, corrected_page_size >= len);
      	r->space_avail = corrected_page_size - len;
      	WT_ASSERT(session, r->space_avail >= add_len);
      	return (0);
      }
      

      All the WT_BOUNDARY.start pointers point into the WT_BUF that were moved as part of __wt_buf_grows. We wind up with a set of pointers in WT_BOUNDARY.start that point to freed memory, because they have not been updated after realloc updates everything.

            Assignee:
            keith.bostic@mongodb.com Keith Bostic (Inactive)
            Reporter:
            keith.bostic@mongodb.com Keith Bostic (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: