Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-9823

Fix race between read and delete of WT_BLOCK in tiered storage

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • WT11.3.0
    • Affects Version/s: None
    • Component/s: None
    • Labels:

      test_tiered08.py failed in an unrelated patch build:

      https://spruce.mongodb.com/task/wiredtiger_ubuntu2004_unit_test_bucket09_patch_fc5838e9095b279533f151b03dc31ff2f3756b00_63121b780305b97e9b422ec1_22_09_02_15_04_25/logs?execution=0

      Here's the stack trace:

      (gdb) bt
      #0  0x00007fc60f96369c in __wt_block_discard (session=session@entry=0x55ba9cad1cd0, 
          block=block@entry=0x55ba9cb69800, added_size=4096) at ../src/block/block_write.c:64
      #1  0x00007fc60f9612d4 in __wt_bm_read (bm=0x55ba9e2ee8c0, session=0x55ba9cad1cd0, 
          buf=0x7ffda74656d0, addr=<optimized out>, addr_size=<optimized out>)
          at ../src/block/block_read.c:44
      #2  0x00007fc60f969066 in __bm_read (bm=<optimized out>, session=<optimized out>, 
          buf=<optimized out>, addr=<optimized out>, addr_size=<optimized out>)
          at ../src/block_cache/block_mgr.c:486
      #3  0x00007fc60f9672d0 in __wt_blkcache_read (session=session@entry=0x55ba9cad1cd0, 
          buf=buf@entry=0x7ffda74656d0, 
          addr=addr@entry=0x7ffda7465701 "\300\206\201\344\a\241 b\001\206\247\375\177", addr_size=10)
          at ../src/block_cache/block_io.c:111
      #4  0x00007fc60f9c1677 in __page_read (session=session@entry=0x55ba9cad1cd0, 
          ref=ref@entry=0x55ba9e1b7360, flags=flags@entry=128) at ../src/btree/bt_read.c:183
      #5  0x00007fc60f9c24aa in __wt_page_in_func (session=session@entry=0x55ba9cad1cd0, 
          ref=ref@entry=0x55ba9e1b7360, flags=flags@entry=128, 
          func=func@entry=0x7fc60fbab9c0 <__PRETTY_FUNCTION__.48765> "__wt_row_search", 
          line=line@entry=438) at ../src/btree/bt_read.c:307
      #6  0x00007fc60fa0bab0 in __wt_page_swap_func (line=438, 
          func=0x7fc60fbab9c0 <__PRETTY_FUNCTION__.48765> "__wt_row_search", flags=128, 
          want=0x55ba9e1b7360, held=0x55ba9e7e6140, session=0x55ba9cad1cd0)
          at ../src/include/btree_inline.h:2150
      #7  __wt_row_search (cbt=cbt@entry=0x55ba9d6ad600, srch_key=srch_key@entry=0x55ba9d6ad718, 
          insert=insert@entry=true, leaf=leaf@entry=0x0, leaf_safe=leaf_safe@entry=false, 
          leaf_foundp=leaf_foundp@entry=0x0) at ../src/btree/row_srch.c:438
      #8  0x00007fc60f98c047 in __cursor_row_search (leaf_foundp=0x0, leaf=0x0, insert=true, 
          cbt=0x55ba9d6ad600) at ../src/btree/bt_cursor.c:501
      #9  __wt_btcur_insert (cbt=cbt@entry=0x55ba9d6ad600) at ../src/btree/bt_cursor.c:1161
      #10 0x00007fc60fa4b7df in __curfile_insert (cursor=0x55ba9d6ad600) at ../src/cursor/cur_file.c:384
      #11 0x00007fc610196306 in _wrap_Cursor_insert (self=<optimized out>, args=<optimized out>)
          at lang/python/wiredtigerPYTHON_wrap.c:4083
      

      Looking at the cause of the segfault:

      (gdb) frame 0
      #0  0x00007fc60f96369c in __wt_block_discard (session=session@entry=0x55ba9cad1cd0, 
          block=block@entry=0x55ba9cb69800, added_size=4096) at ../src/block/block_write.c:64
      64	    handle = block->fh->handle;
      (gdb) print block
      $4 = (WT_BLOCK *) 0x55ba9cb69800
      (gdb) print block->fh
      $5 = (WT_FH *) 0xabababababababab
      (gdb) print *block
      $6 = {name = 0x0, objectid = 2880154539, ref = 2880154539, q = {tqe_next = 0xabababababababab, 
          tqe_prev = 0xabababababababab}, hashq = {tqe_next = 0xabababababababab, 
      
      ...

      In other words we are trying to use block after it has been freed. This appears to be a race with whoever freed it. This function was called from __wt_bm_read() here:

          /* Read the block. */
          __wt_capacity_throttle(session, size, WT_THROTTLE_READ);
          WT_RET(__wt_block_read_off(session, block, buf, objectid, offset, size, checksum));
      
          /* Optionally discard blocks from the system's buffer cache. */
          WT_RET(__wt_block_discard(session, block, (size_t)size));

      In other words, by the time we get into __wt_block_discard(), block has been freed. But we successfully completed the previous call into __wt_block_read_off() which used block. So some other thread must have freed it in the interim.

      Note that this patch includes WT-7833.  

       

            Assignee:
            keith.smith@mongodb.com Keith Smith
            Reporter:
            keith.smith@mongodb.com Keith Smith
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: