Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-1239

format core dump 5353, WT_BTREE.compressor == NULL

    • Type: Icon: Task Task
    • Resolution: Done
    • WT2.4.0
    • Affects Version/s: None
    • Component/s: None

      Format core dump 5353, here's the CONFIG:

      ############################################
      #  RUN PARAMETERS
      # stress run WT-5353
      ############################################
      auto_throttle=1
      firstfit=0
      bitcnt=3
      bloom=1
      bloom_bit_count=45
      bloom_hash_count=20
      bloom_oldest=0
      cache=59
      checkpoints=1
      checksum=off
      chunk_size=1
      compaction=0
      compression=zlib
      data_extend=0
      data_source=file
      delete_pct=42
      dictionary=0
      evict_max=5
      file_type=variable-length column-store
      backups=0
      huffman_key=0
      huffman_value=0
      insert_pct=11
      internal_key_truncation=1
      internal_page_max=10
      isolation=read-uncommitted
      key_gap=16
      key_max=49
      key_min=13
      leak_memory=0
      leaf_page_max=17
      logging=1
      merge_max=17
      merge_threads=2
      mmap=1
      ops=100000
      prefix_compression=1
      prefix_compression_min=2
      repeat_data_pct=15
      reverse=0
      rows=100000
      runs=100
      split_pct=65
      statistics=0
      threads=8
      value_max=209
      value_min=0
      wiredtiger_config=
      write_pct=41
      ############################################
      

      I've got thousands of runs of this without a failure, so it's not an obvious failure.
      Here's the stack:

      (gdb) where
      #0  0x00000000004bc91d in __wt_bt_write (session=0x251e400, 
          buf=0x7fd71819eb58, addr=0x7fd757ffe6f0 "\030", addr_sizep=0x7fd757ffe7f8, 
          checkpoint=0, compressed=1) at ../src/btree/bt_io.c:160
      WT-1  0x00000000004715f8 in __rec_split_write (session=0x251e400, 
          r=0x7fd71819eb40, bnd=0x7fd7180b6ce0, buf=0x7fd71819eb58, last_block=1)
          at ../src/btree/rec_write.c:2786
      WT-2  0x0000000000470ba9 in __rec_split_finish_std (session=0x251e400, 
          r=0x7fd71819eb40) at ../src/btree/rec_write.c:2501
      WT-3  0x0000000000470c71 in __rec_split_finish (session=0x251e400, 
          r=0x7fd71819eb40) at ../src/btree/rec_write.c:2532
      WT-4  0x000000000047527b in __rec_row_leaf (session=0x251e400, r=0x7fd71819eb40, 
          page=0x2554cd0, salvage=0x0) at ../src/btree/rec_write.c:4493
      WT-5  0x000000000046d085 in __wt_rec_write (session=0x251e400, ref=0x2554c90, 
          salvage=0x0, flags=0) at ../src/btree/rec_write.c:411
      WT-6  0x00000000004605b4 in __sync_file (session=0x251e400, syncop=8)
          at ../src/btree/bt_sync.c:135
      WT-7  0x0000000000460aa7 in __wt_cache_op (session=0x251e400, 
          ckptbase=0x7fd71804dd00, op=8) at ../src/btree/bt_sync.c:354
      WT-8  0x000000000044b4e4 in __checkpoint_worker (session=0x251e400, 
          cfg=0x7fd757ffed70, is_checkpoint=1) at ../src/txn/txn_ckpt.c:841
      WT-9  0x000000000044b72f in __wt_checkpoint (session=0x251e400, 
          cfg=0x7fd757ffed70) at ../src/txn/txn_ckpt.c:899
      WT-10 0x000000000044a620 in __wt_txn_checkpoint (session=0x251e400, 
          cfg=0x7fd757ffed70) at ../src/txn/txn_ckpt.c:402
      WT-11 0x0000000000440b1d in __session_checkpoint (wt_session=0x251e400, 
          config=0x0) at ../src/session/session_api.c:793
      WT-12 0x0000000000406cc5 in ops (arg=0x2544600) at ../../../test/format/ops.c:289
      WT-13 0x000000339e807c53 in start_thread (arg=0x7fd757fff700)
          at pthread_create.c:308
      WT-14 0x000000339e0f5dbd in clone ()
          at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
      (gdb) frame 0
      #0  0x00000000004bc91d in __wt_bt_write (session=0x251e400, 
          buf=0x7fd71819eb58, addr=0x7fd757ffe6f0 "\030", addr_sizep=0x7fd757ffe7f8, 
          checkpoint=0, compressed=1) at ../src/btree/bt_io.c:160
      160			WT_ERR(btree->compressor->decompress(
      (gdb) p btree->compressor
      $12 = (WT_COMPRESSOR *) 0x0
      (gdb) p btree->dhandle->name
      $13 = 0x2548bf0 "file:WiredTiger.wt"
      (gdb) frame 1
      WT-1  0x00000000004715f8 in __rec_split_write (session=0x251e400, 
          r=0x7fd71819eb40, bnd=0x7fd7180b6ce0, buf=0x7fd71819eb58, last_block=1)
          at ../src/btree/rec_write.c:2786
      2786		WT_ERR(__wt_bt_write(session,
      (gdb) p *bnd
      $14 = {start = 0x7fd71819ee28 "!file:wt", recno = 0, entries = 2, addr = {
          addr = 0x0, size = 0 '\000', type = 3 '\003', reuse = 0 '\000'}, 
        size = 1629, cksum = 0, dsk = 0x0, skip = 0x0, skip_next = 0, 
        skip_allocated = 0, key = {data = 0x0, size = 0, flags = 0, mem = 0x0, 
          memsize = 0}, already_compressed = 1}
      

      This is diagnostic code that decompresses and verifies compressed buffers before writing them:

      #ifdef HAVE_DIAGNOSTIC
              /*
               * We're passed a table's disk image.  Decompress if necessary and
               * verify the image.  Always check the in-memory length for accuracy.
               */
              dsk = buf->mem;
              if (compressed) {
                      WT_ERR(__wt_scr_alloc(session, dsk->mem_size, &tmp));
                      
                      memcpy(tmp->mem, buf->data, WT_BLOCK_COMPRESS_SKIP);
                      WT_ERR(btree->compressor->decompress(
      

      What happened is we're writing the WiredTiger.wt file (which isn't compressed), but the WT_BOUNDARY we're using to write the block has the WT_BOUNDARY.already_compressed flag set, presumably from a previous use of the WT_BOUNDARY structure to write a compressed block. (It must be from a previous run: WT_BOUNDARY.already_compressed is only set in *rec_split_raw_worker(), and we're not going through the raw-compression code here, the call to *rec_split_finish_std() tells us this reconciliation isn't configured for raw-compression.

      I can make this bug happen by explicitly not resetting WT_BOUNDARY.already_compressed inside __rec_bnd_cleanup(), but I don't yet see any other path where we might end up with it not reset.

            Assignee:
            keith.bostic@mongodb.com Keith Bostic (Inactive)
            Reporter:
            keith.bostic@mongodb.com Keith Bostic (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: