Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-2985

Race during checkpoint can cause a core dump

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • WT2.9.0, 3.2.11, 3.4.0-rc2
    • Affects Version/s: None
    • Component/s: None
    • Labels:
      None
    • Storage 2016-10-31

      Running in HAVE_DIAGNOSTIC mode, the failure is as follows:

      file:test00000.wt, WT_SESSION.checkpoint: src/reconcile/rec_write.c, 6061: size != 0 || cell_type == WT_CELL_ADDR_DEL
      file:test00000.wt, WT_SESSION.checkpoint: aborting WiredTiger library
      file:test00000.wt, WT_SESSION.checkpoint: process ID 23234: waiting for debugger...
      
      (gdb) where
      #0  0x00007fc5e378e3c3 in select () from /lib64/libc.so.6
      #1  0x00000000004526bc in __wt_sleep (seconds=10, micro_seconds=0) at src/os_posix/os_sleep.c:23
      #2  0x00000000004897dc in __wt_attach (session=0x10bcb40) at src/support/global.c:125
      #3  0x000000000051e3e2 in __wt_abort (session=0x10bcb40) at src/os_common/os_abort.c:22
      #4  0x000000000048933d in __wt_assert (session=0x10bcb40, error=0, file_name=0x561479 "src/reconcile/rec_write.c", line_number=6061, fmt=0x56126a "%s") at src/support/err.c:482
      #5  0x000000000046ac42 in __rec_cell_build_addr (session=0x10bcb40, r=0x7fc0e43b9920, addr=0x0, size=0, cell_type=48, recno=0) at src/reconcile/rec_write.c:6061
      #6  0x00000000004685ef in __rec_row_merge (session=0x10bcb40, r=0x7fc0e43b9920, page=0x7fc57d0b8b70) at src/reconcile/rec_write.c:4983
      #7  0x0000000000468014 in __rec_row_int (session=0x10bcb40, r=0x7fc0e43b9920, page=0x7fc5aceef890) at src/reconcile/rec_write.c:4857
      #8  0x000000000045f79a in __wt_reconcile (session=0x10bcb40, ref=0x7fc5c404c2a0, salvage=0x0, flags=1) at src/reconcile/rec_write.c:415
      #9  0x00000000004d04d8 in __sync_file (session=0x10bcb40, syncop=WT_SYNC_CHECKPOINT) at src/btree/bt_sync.c:197
      #10 0x00000000004d06c4 in __wt_cache_op (session=0x10bcb40, op=WT_SYNC_CHECKPOINT) at src/btree/bt_sync.c:277
      #11 0x000000000049c170 in __checkpoint_tree (session=0x10bcb40, is_checkpoint=true, cfg=0x7fc5e117fe70) at src/txn/txn_ckpt.c:1390
      #12 0x000000000049c432 in __checkpoint_tree_helper (session=0x10bcb40, cfg=0x7fc5e117fe70) at src/txn/txn_ckpt.c:1497
      #13 0x0000000000498426 in __checkpoint_apply (session=0x10bcb40, cfg=0x7fc5e117fe70, op=0x49c3f3 <__checkpoint_tree_helper>) at src/txn/txn_ckpt.c:190
      #14 0x000000000049a691 in __txn_checkpoint (session=0x10bcb40, cfg=0x7fc5e117fe70) at src/txn/txn_ckpt.c:731
      #15 0x000000000049b2ae in __wt_txn_checkpoint (session=0x10bcb40, cfg=0x7fc5e117fe70) at src/txn/txn_ckpt.c:916
      #16 0x0000000000483e25 in __session_checkpoint (wt_session=0x10bcb40, config=0x0) at src/session/session_api.c:1579
      

      The problem is we're building an address that will reference a leaf page, but the leaf page has a disk image instead of an address.

      To reproduce the problem, apply this diff:

      diff --git a/bench/wtperf/runners/500m-btree-50r50u.wtperf b/bench/wtperf/runners/500m-btree-50r50u.wtperf
      index 06745bf..452ce43 100644
      --- a/bench/wtperf/runners/500m-btree-50r50u.wtperf
      +++ b/bench/wtperf/runners/500m-btree-50r50u.wtperf
      @@ -7,7 +7,7 @@
       # checkpoints.  Collect wiredtiger stats for ftdc.
       conn_config="cache_size=16G,checkpoint=(wait=60,log_size=2GB),session_max=20000,log=(enabled),statistics=(fast),statistics_log=(wait=30,json),eviction=(threads_max=4)"
       create=false
      -compression="snappy"
      +compression="zlib"
       sess_config="isolation=snapshot"
       table_count=2
       key_sz=40
      diff --git a/bench/wtperf/runners/500m-btree-populate.wtperf b/bench/wtperf/runners/500m-btree-populate.wtperf
      index f9aed09..170ea0e 100644
      --- a/bench/wtperf/runners/500m-btree-populate.wtperf
      +++ b/bench/wtperf/runners/500m-btree-populate.wtperf
      @@ -11,7 +11,7 @@
       # well and be small on disk.
       conn_config="cache_size=16G,checkpoint=(wait=60,log_size=2GB),session_max=20000,log=(enabled),statistics=(fast),statistics_log=(wait=30,json),eviction=(threads_max=4)"
       compact=true
      -compression="snappy"
      +compression="zlib"
       sess_config="isolation=snapshot"
       table_config="internal_page_max=16K,type=file,leaf_page_max=16K,memory_page_max=10M,split_pct=90"
       table_count=2
      

      Then run 500m-btree-populate.wtperf followed by 500m-btree-50r50u.wtperf. It hits pretty reliably for me, but can take anywhere from 3-4 runs, and 30 minutes to an hour for each run.

            Assignee:
            keith.bostic@mongodb.com Keith Bostic (Inactive)
            Reporter:
            keith.bostic@mongodb.com Keith Bostic (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: