-
Type:
Build Failure
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
Storage Engines, Storage Engines - Persistence
-
SE Persistence backlog
-
None
-
6
Host: rhel8-medium-6.z.devprod.mongodb.com
Project: wiredtiger
Commit: f309fb09
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules
Task Logs:
unit-test task_log
Logs:
[pid:855151]: test_layered47.test_layered47.test_checkpoint_order_mismatch -s 0 (palm): starting [pid:855151]: test_layered47.test_layered47.test_checkpoint_order_mismatch -s 0 (palm): 0.18 seconds [pid:855149/855151]: test_layered47.test_layered47.test_checkpoint_order_mismatch(palm) ... ok
unit-test task_log
Logs:
[pid:855149]: ERROR: test_layered39.test_layered39.test_layered39(palm) ---------------------------------------------------------------------- [pid:855149]: testtools.testresult.real._StringException: lost connection during test 'test_layered39.test_layered39.test_layered39(palm)' ---------------------------------------------------------------------- Ran 10766 tests in 7551.955s
unit-test task_log
Logs:
#0 0x000003ffbafc71e4 in raise () from /lib64/libc.so.6 #0 0x000003ffbafc71e4 in raise () from /lib64/libc.so.6 #1 0x000003ffbafa3308 in abort () from /lib64/libc.so.6 #2 0x000003ffad13d792 in __wt_abort (session=session@entry=0xf12fed8) at /data/mci/65e2935bb2e91a05ccf89f8c1f1463ab/wiredtiger/src/os_common/os_abort.c:32 #3 0x000003ffad218bae in __wt_txn_rollback (session=session@entry=0xf12fed8, cfg=cfg@entry=0x0, api_call=api_call@entry=false) at /data/mci/65e2935bb2e91a05ccf89f8c1f1463ab/wiredtiger/src/txn/txn.c:2079 #4 0x000003ffad039f8e in __checkpoint_db_internal (session=session@entry=0xf12fed8, cfg=cfg@entry=0x3ffee67b588) at /data/mci/65e2935bb2e91a05ccf89f8c1f1463ab/wiredtiger/src/checkpoint/checkpoint_txn.c:1643 #5 0x000003ffad03a62c in __checkpoint_db_wrapper (session=session@entry=0xf12fed8, cfg=cfg@entry=0x3ffee67b588) at /data/mci/65e2935bb2e91a05ccf89f8c1f1463ab/wiredtiger/src/checkpoint/checkpoint_txn.c:1743 #6 0x000003ffad03a9e0 in __wt_checkpoint_db (session=0xf12fed8, cfg=cfg@entry=0x3ffee67b588, waiting=waiting@entry=true) at /data/mci/65e2935bb2e91a05ccf89f8c1f1463ab/wiredtiger/src/checkpoint/checkpoint_txn.c:1822 #7 0x000003ffad21745e in __wt_txn_global_shutdown (session=session@entry=0xf12a780, cfg=cfg@entry=0x3ffee67b6c0) at /data/mci/65e2935bb2e91a05ccf89f8c1f1463ab/wiredtiger/src/txn/txn.c:2557 #8 0x000003ffad050d4c in __conn_close (wt_conn=0xff16c50, config=<optimized out>) at /data/mci/65e2935bb2e91a05ccf89f8c1f1463ab/wiredtiger/src/conn/conn_api.c:1241 #9 0x000003ffad417832 in _wrap_Connection_close (self=<optimized out>, args=<optimized out>) at /data/mci/65e2935bb2e91a05ccf89f8c1f1463ab/wiredtiger/cmake_build/lang/python/CMakeFiles/wiredtiger_python.dir/wiredtigerPYTHON_wrap.c:8659 #10 0x000003ffbb5db7fe in cfunction_call (func=0x3ffad7c3010, args=<optimized out>, kwargs=<optimized out>) at ../src/Python-3.10.4/Objects/methodobject.c:552 #11 0x000003ffbb5ed71c in _PyObject_Call (tstate=0xc6baf30, callable=0x3ffad7c3010, args=0x3ff3d047310, kwargs=0x0) at ../src/Python-3.10.4/Objects/call.c:305 #12 PyObject_Call (callable=0x3ffad7c3010, args=0x3ff3d047310, kwargs=0x0) at ../src/Python-3.10.4/Objects/call.c:317 #13 0x000003ffbb5c7992 in do_call_core (tstate=<optimized out>, trace_info=0x3ffee67beb8, func=0x3ffad7c3010, callargs=0x3ff3d047310, kwdict=0x0) at ../src/Python-3.10.4/Python/ceval.c:5919 #14 _PyEval_EvalFrameDefault (tstate=tstate@entry=0xc6baf30, f=f@entry=0x3ffaa8be0e0, throwflag=throwflag@entry=0) at ../src/Python-3.10.4/Python/ceval.c:4277 #15 0x000003ffbb5dbeca in _PyEval_EvalFrame (tstate=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=0x3ffaa8be0e0, throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at ../src/Python-3.10.4/Python/ceval.c:5052 #16 _PyEval_Vector (tstate=0xc6baf30, con=0x3ffad73bd10, locals=0x0, args=0xccf8230, argcount=4396895026890, kwnames=<optimized out>) at ../src/Python-3.10.4/Python/ceval.c:5065 #17 _PyFunction_Vectorcall (func=func@entry=0x3ffad73bd00, stack=stack@entry=0xccf8230, nargsf=nargsf@entry=9223376433749709216, kwnames=kwnames@entry=0x0) at ../src/Python-3.10.4/Objects/call.c:342 #18 0x000003ffbb5c51a0 in _PyObject_VectorcallTstate (tstate=0xc6baf30, callable=0x3ffad73bd00, args=0xccf8230, nargsf=9223376433749709216, kwnames=0x0) at ../src/Python-3.10.4/Include/cpython/abstract.h:123 #19 PyObject_Vectorcall (callable=<optimized out>, args=0xccf8230, nargsf=9223376433749709216, kwnames=0x0) at ../src/Python-3.10.4/Include/cpython/abstract.h:123 #20 call_function (tstate=<optimized out>, trace_info=0x3ffee67c158, pp_stack=<synthetic pointer>, oparg=4396894933408, kwnames=0x0) at ../src/Python-3.10.4/Python/ceval.c:5867 #21 _PyEval_EvalFrameDefault (tstate=tstate@entry=0xc6baf30, f=f@entry=0xccf8070, throwflag=throwflag@entry=0) at ../src/Python-3.10.4/Python/ceval.c:4198 #22 0x000003ffbb5dbeca in _PyEval_EvalFrame (tstate=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=0xccf8070, throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at ../src/Python-3.10.4/Python/ceval.c:5052 #23 _PyEval_Vector (tstate=0xc6baf30, con=0x3ffacd55250, locals=0x0, args=0x3ffaa8fcfc0, argcount=4396895026890, kwnames=<optimized out>) at ../src/Python-3.10.4/Python/ceval.c:5065 #24 _PyFunction_Vectorcall (func=func@entry=0x3ffacd55240, stack=stack@entry=0x3ffaa8fcfc0, nargsf=nargsf@entry=9223376433749709216, kwnames=kwnames@entry=0x0) at ../src/Python-3.10.4/Objects/call.c:342 #25 0x000003ffbb5c51a0 in _PyObject_VectorcallTstate (tstate=0xc6baf30, callable=0x3ffacd55240, args=0x3ffaa8fcfc0, nargsf=9223376433749709216, kwnames=0x0) at ../src/Python-3.10.4/Include/cpython/abstract.h:123 #26 PyObject_Vectorcall (callable=<optimized out>, args=0x3ffaa8fcfc0, nargsf=9223376433749709216, kwnames=0x0) at ../src/Python-3.10.4/Include/cpython/abstract.h:123 #27 call_function (tstate=<optimized out>, trace_info=0x3ffee67c3f8, pp_stack=<synthetic pointer>, oparg=4396894933408, kwnames=0x0) at ../src/Python-3.10.4/Python/ceval.c:5867 #28 _PyEval_EvalFrameDefault (tstate=tstate@entry=0xc6baf30, f=f@entry=0x3ffaa8fce50, throwflag=throwflag@entry=0) at ../src/Python-3.10.4/Python/ceval.c:4198 #29 0x000003ffbb5dbeca in _PyEval_EvalFrame (tstate=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=0x3ffaa8fce50, throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at ../src/Python-3.10.4/Python/ceval.c:5052
unit-test task_log
Logs:
[1760443110:546642][855150:0x3ffbb975440], test_layered39.test_layered39.test_layered39(palm), file:WiredTigerShared.wt_stable, WT_CURSOR.insert: [WT_VERB_DEFAULT][ERROR]: __block_disagg_check_lsn_frontier, 74: Attempted to read page with LSN 1489 ahead of the materialization frontier at LSN 10: Invalid argument [1760443110:546812][855150:0x3ffbb975440], test_layered39.test_layered39.test_layered39(palm), close_ckpt: [WT_VERB_DEFAULT][ERROR]: __wt_txn_rollback, 2079: WiredTiger assertion failed: '!(((((op->btree->dhandle))->flags) & (0x080u)) != 0)'. Expression returned false [1760443110:546825][855150:0x3ffbb975440], test_layered39.test_layered39.test_layered39(palm), close_ckpt: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library [1760443110:546838][855150:0x3ffbb975440], test_layered39.test_layered39.test_layered39(palm), close_ckpt: [WT_VERB_ERROR_RETURNS][ERROR]: __cursors_can_be_cached, 922: Error at src/cursor/cur_std.c:922: "__wt_config_gets(session, cfg, "force", &cval)" failed: WT_NOTFOUND: item not found [1760443110:546851][855150:0x3ffbb975440], test_layered39.test_layered39.test_layered39(palm), close_ckpt: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_cursor_cache_get, 1063: Error at src/cursor/cur_std.c:1063: "__cursors_can_be_cached(session, cfg, &cacheable)" failed: WT_NOTFOUND: item not found [1760443110:546860][855150:0x3ffbb975440], test_layered39.test_layered39.test_layered39(palm), close_ckpt: [WT_VERB_ERROR_RETURNS][ERROR]: __block_disagg_check_lsn_frontier, 74: Error at src/block_disagg/block_disagg_read.c:74: "EINVAL" failed with WT_DISAGG_BAD_LSN: Attempt to operate on a wrong LSN number: Invalid argument [1760443110:546870][855150:0x3ffbb975440], test_layered39.test_layered39.test_layered39(palm), close_ckpt: [WT_VERB_ERROR_RETURNS][ERROR]: __block_disagg_read_multiple, 117: Error at src/block_disagg/block_disagg_read.c:117: "__block_disagg_check_lsn_frontier(session, lsn)" failed: Invalid argument [1760443110:546879][855150:0x3ffbb975440], test_layered39.test_layered39.test_layered39(palm), close_ckpt: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_block_disagg_read_multiple, 317: Error at src/block_disagg/block_disagg_read.c:317: "__block_disagg_read_multiple(session, block_disagg, block_meta, cookie.page_id, cookie.flags, cookie.lsn, cookie.base_lsn, cookie.size, cookie.checksum, buffer_array, buffer_count)" failed: Invalid argument [1760443110:546890][855150:0x3ffbb975440], test_layered39.test_layered39.test_layered39(palm), close_ckpt: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_blkcache_read_multi, 472: Error at src/block_cache/block_io.c:472: "bm->read_multiple(bm, session, &block_meta_tmp, addr, addr_size, &results[0], &count)" failed: Invalid argument [1760443110:546899][855150:0x3ffbb975440], test_layered39.test_layered39.test_layered39(palm), close_ckpt: [WT_VERB_ERROR_RETURNS][ERROR]: __page_read, 235: Error at src/btree/bt_read.c:235: "__wt_blkcache_read_multi(session, &tmp, &count, &block_meta, addr.addr, addr.size)" failed: Invalid argument [1760443110:546907][855150:0x3ffbb975440], test_layered39.test_layered39.test_layered39(palm), close_ckpt: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_page_in_func, 419: Error at src/btree/bt_read.c:419: "__page_read(session, ref, flags)" failed: Invalid argument [1760443110:546916][855150:0x3ffbb975440], test_layered39.test_layered39.test_layered39(palm), close_ckpt: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_btcur_insert, 1193: Error at src/btree/bt_cursor.c:1193: "__cursor_row_search(cbt, true, NULL, NULL)" failed: Invalid argument [1760443110:546924][855150:0x3ffbb975440], test_layered39.test_layered39.test_layered39(palm), close_ckpt: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_insert, 389: Error at src/cursor/cur_file.c:389: "__wt_btcur_insert(cbt)" failed: Invalid argument [1760443110:546934][855150:0x3ffbb975440], test_layered39.test_layered39.test_layered39(palm), close_ckpt: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_disagg_update_shared_metadata, 2064: Error at src/conn/conn_layered.c:2064: "cursor->insert(cursor)" failed: Invalid argument [1760443110:546942][855150:0x3ffbb975440], test_layered39.test_layered39.test_layered39(palm), close_ckpt: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_block_disagg_checkpoint_resolve, 152: Error at src/block_disagg/block_disagg_ckpt.c:152: "ret" failed: Invalid argument [1760443110:546953][855150:0x3ffbb975440], test_layered39.test_layered39.test_layered39(palm), close_ckpt: [WT_VERB_ERROR_RETURNS][ERROR]: __checkpoint_tree, 2627: Error at src/checkpoint/checkpoint_txn.c:2627: "bm->checkpoint_resolve(bm, session, ret != 0)" failed: Invalid argument [1760443110:546961][855150:0x3ffbb975440], test_layered39.test_layered39.test_layered39(palm), close_ckpt: [WT_VERB_ERROR_RETURNS][ERROR]: __checkpoint_apply_to_dhandles, 341: Error at src/checkpoint/checkpoint_txn.c:341: "ret" failed: Invalid argument [1760443110:546970][855150:0x3ffbb975440], test_layered39.test_layered39.test_layered39(palm), close_ckpt: [WT_VERB_ERROR_RETURNS][ERROR]: __checkpoint_db_internal, 1349: Error at src/checkpoint/checkpoint_txn.c:1349: "__checkpoint_apply_to_dhandles(session, cfg, __checkpoint_tree_helper)" failed: Invalid argument