Major - P3 There are many failures across MongoDB that are failing on the following assert in __wt_hs_insert_updates:
WT_ASSERT(session, out_of_order_ts_updates.size == 0);
This assert was added in WT-7229, and the first BFG occurred on the WT drop that included this ticket's most recent commit.
The pretty-printed stack trace:
(gdb) bt #0 0x00007f56787e293f in raise () from /lib64/libc.so.6 #1 0x00007f56787ccc95 in abort () from /lib64/libc.so.6 #2 0x00007f566e8be39f in __wt_abort (session=session@entry=0x7f5667bbcee0) at src/third_party/wiredtiger/src/os_common/os_abort.c:30 #3 0x00007f566e8b747d in __wt_hs_insert_updates (session=session@entry=0x7f5667bbcee0, page=0x7f566c6c9820, multi=multi@entry=0x7f563de61a20, cache_write_hs=cache_write_hs@entry=0x7f563f2f0930) at src/third_party/wiredtiger/src/history/hs_rec.c:359 #4 0x00007f566ea3eff3 in __rec_hs_wrapup (r=0x7f563f2f0420, session=0x7f5667bbcee0) at src/third_party/wiredtiger/src/reconcile/rec_write.c:2294 #5 __rec_write_wrapup (session=0x7f5667bbcee0, r=0x7f563f2f0420, page=0x7f566c6c9820) at src/third_party/wiredtiger/src/reconcile/rec_write.c:2120 #6 0x00007f566ea40801 in __reconcile (page_lockedp=<synthetic pointer>, flags=36, salvage=0x0, ref=0x7f564037d1e0, session=0x7f5667bbcee0) at src/third_party/wiredtiger/src/reconcile/rec_write.c:213 #7 __wt_reconcile (session=session@entry=0x7f5667bbcee0, ref=0x7f564037d1e0, salvage=salvage@entry=0x0, flags=flags@entry=36) at src/third_party/wiredtiger/src/reconcile/rec_write.c:99 #8 0x00007f566e93c1b2 in __wt_sync_file (session=session@entry=0x7f5667bbcee0, syncop=syncop@entry=WT_SYNC_CHECKPOINT) at src/third_party/wiredtiger/src/btree/bt_sync.c:642 #9 0x00007f566ea946ec in __checkpoint_tree (session=0x7f5667bbcee0, is_checkpoint=<optimized out>, cfg=<optimized out>) at src/third_party/wiredtiger/src/txn/txn_ckpt.c:1728 #10 0x00007f566ea95647 in __checkpoint_tree_helper (cfg=0x7f565e099910, session=0x7f5667bbcee0) at src/third_party/wiredtiger/src/txn/txn_ckpt.c:1836 #11 __checkpoint_apply_to_dhandles (op=<optimized out>, cfg=0x7f565e099910, session=0x7f5667bbcee0) at src/third_party/wiredtiger/src/txn/txn_ckpt.c:197 #12 __txn_checkpoint (cfg=<optimized out>, session=0x7f5667bbcee0) at src/third_party/wiredtiger/src/txn/txn_ckpt.c:880 #13 __txn_checkpoint_wrapper (session=0x7f5667bbcee0, cfg=<optimized out>) at src/third_party/wiredtiger/src/txn/txn_ckpt.c:1109 #14 0x00007f566ea96d96 in __wt_txn_checkpoint (session=session@entry=0x7f5667bbcee0, cfg=cfg@entry=0x7f565e099910, waiting=waiting@entry=true) at src/third_party/wiredtiger/src/txn/txn_ckpt.c:1163 #15 0x00007f566e8c7a35 in __session_checkpoint (wt_session=0x7f5667bbcee0, config=0x7f5671e2d7a8 "use_timestamp=true") at src/third_party/wiredtiger/src/session/session_api.c:1890 #16 0x00007f5671de7a14 in mongo::WiredTigerKVEngine::checkpoint (this=0x7f566c51b820) at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp:1849
The value of out_of_order_ts_updates:
(gdb) p out_of_order_ts_updates
$22 = {
session = 0x7f5667bbcee0,
list = {0x7f56402c2100, 0x0 <repeats 19 times>},
listp = 0x7f565e098d38,
allocated_bytes = 0,
size = 1
}
