Fix race between read and delete of WT_BLOCK in tiered storage

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fixed
    • Priority: Major - P3
    • WT11.3.0
    • Affects Version/s: None
    • Component/s: None

      test_tiered08.py failed in an unrelated patch build:

      https://spruce.mongodb.com/task/wiredtiger_ubuntu2004_unit_test_bucket09_patch_fc5838e9095b279533f151b03dc31ff2f3756b00_63121b780305b97e9b422ec1_22_09_02_15_04_25/logs?execution=0

      Here's the stack trace:

      (gdb) bt
#0  0x00007fc60f96369c in __wt_block_discard (session=session@entry=0x55ba9cad1cd0, 
    block=block@entry=0x55ba9cb69800, added_size=4096) at ../src/block/block_write.c:64
#1  0x00007fc60f9612d4 in __wt_bm_read (bm=0x55ba9e2ee8c0, session=0x55ba9cad1cd0, 
    buf=0x7ffda74656d0, addr=<optimized out>, addr_size=<optimized out>)
    at ../src/block/block_read.c:44
#2  0x00007fc60f969066 in __bm_read (bm=<optimized out>, session=<optimized out>, 
    buf=<optimized out>, addr=<optimized out>, addr_size=<optimized out>)
    at ../src/block_cache/block_mgr.c:486
#3  0x00007fc60f9672d0 in __wt_blkcache_read (session=session@entry=0x55ba9cad1cd0, 
    buf=buf@entry=0x7ffda74656d0, 
    addr=addr@entry=0x7ffda7465701 "\300\206\201\344\a\241 b\001\206\247\375\177", addr_size=10)
    at ../src/block_cache/block_io.c:111
#4  0x00007fc60f9c1677 in __page_read (session=session@entry=0x55ba9cad1cd0, 
    ref=ref@entry=0x55ba9e1b7360, flags=flags@entry=128) at ../src/btree/bt_read.c:183
#5  0x00007fc60f9c24aa in __wt_page_in_func (session=session@entry=0x55ba9cad1cd0, 
    ref=ref@entry=0x55ba9e1b7360, flags=flags@entry=128, 
    func=func@entry=0x7fc60fbab9c0 <__PRETTY_FUNCTION__.48765> "__wt_row_search", 
    line=line@entry=438) at ../src/btree/bt_read.c:307
#6  0x00007fc60fa0bab0 in __wt_page_swap_func (line=438, 
    func=0x7fc60fbab9c0 <__PRETTY_FUNCTION__.48765> "__wt_row_search", flags=128, 
    want=0x55ba9e1b7360, held=0x55ba9e7e6140, session=0x55ba9cad1cd0)
    at ../src/include/btree_inline.h:2150
#7  __wt_row_search (cbt=cbt@entry=0x55ba9d6ad600, srch_key=srch_key@entry=0x55ba9d6ad718, 
    insert=insert@entry=true, leaf=leaf@entry=0x0, leaf_safe=leaf_safe@entry=false, 
    leaf_foundp=leaf_foundp@entry=0x0) at ../src/btree/row_srch.c:438
#8  0x00007fc60f98c047 in __cursor_row_search (leaf_foundp=0x0, leaf=0x0, insert=true, 
    cbt=0x55ba9d6ad600) at ../src/btree/bt_cursor.c:501
#9  __wt_btcur_insert (cbt=cbt@entry=0x55ba9d6ad600) at ../src/btree/bt_cursor.c:1161
#10 0x00007fc60fa4b7df in __curfile_insert (cursor=0x55ba9d6ad600) at ../src/cursor/cur_file.c:384
#11 0x00007fc610196306 in _wrap_Cursor_insert (self=<optimized out>, args=<optimized out>)
    at lang/python/wiredtigerPYTHON_wrap.c:4083

      Looking at the cause of the segfault:

      (gdb) frame 0
#0  0x00007fc60f96369c in __wt_block_discard (session=session@entry=0x55ba9cad1cd0, 
    block=block@entry=0x55ba9cb69800, added_size=4096) at ../src/block/block_write.c:64
64	    handle = block->fh->handle;
(gdb) print block
$4 = (WT_BLOCK *) 0x55ba9cb69800
(gdb) print block->fh
$5 = (WT_FH *) 0xabababababababab
(gdb) print *block
$6 = {name = 0x0, objectid = 2880154539, ref = 2880154539, q = {tqe_next = 0xabababababababab, 
    tqe_prev = 0xabababababababab}, hashq = {tqe_next = 0xabababababababab, 

...

      In other words we are trying to use block after it has been freed. This appears to be a race with whoever freed it. This function was called from __wt_bm_read() here:

          /* Read the block. */
    __wt_capacity_throttle(session, size, WT_THROTTLE_READ);
    WT_RET(__wt_block_read_off(session, block, buf, objectid, offset, size, checksum));

    /* Optionally discard blocks from the system's buffer cache. */
    WT_RET(__wt_block_discard(session, block, (size_t)size));

      In other words, by the time we get into __wt_block_discard(), block has been freed. But we successfully completed the previous call into __wt_block_read_off() which used block. So some other thread must have freed it in the interim.

      Note that this patch includes WT-7833.  

       

            Assignee:
            Keith Smith
            Reporter:
            Keith Smith
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: