Loading...

XML

Word

Printable

JSON

Type: Build Failure
Resolution: Unresolved
Priority: Major - P3
Fix Version/s: None
Affects Version/s: None
Component/s: Verify
Labels:
- BB-Tools

Assigned Teams:

Storage Engines - Transactions
Total Hours with Assigned Team:
134.032
Sprint:
SE Transactions - 2026-08-14
Story Points:
3
Evergreen Project:
- wiredtiger
Failure Type:
- failed
Failing Buildvariants:
- rhel8-ppc
- ubuntu2004-release
Failing Tasks:
First Failing Revision:
- 05921ca764dc51206fa95fe3274e81a66fb8774b(wiredtiger)
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-18085
Count of Linked BFGs (Last 30 days):
3

format-test on ubuntu2004-release

Host: i-01b0f3222bc38ee30
Project: wiredtiger
Commit: 05921ca7
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

Task Logs:

task format-test

format-test task_log

Logs:

[398/680] Building C object test/csuite/CMakeFiles/test_wt16990_disagg_checkpoint_panic.dir/wt16990_disagg_checkpoint_panic/main.c.o
[602/680] Linking CXX executable test/csuite/wt16990_disagg_checkpoint_panic/test_wt16990_disagg_checkpoint_panic

logs

format-test task_log

Logs:

1/1 Test #1: test_format ......................***Failed  163.09 sec

logs

format-test task_log

Logs:

[1784029357:181193][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_key_hs, 1431: key 88987 has an overlap of timestamp ranges between history store stop timestamp (0, 756280) being newer than a more recent timestamp range having start timestamp (0, 622547): WT_ERROR: non-specific WiredTiger error
[1784029357:191662][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 2048-2560 never verified
[1784029357:191700][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 3072-3584 never verified
[1784029357:191706][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 9728-10240 never verified
[1784029357:191710][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 116224-116736 never verified
[1784029357:191714][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 121344-121856 never verified
[1784029357:191718][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 140800-141312 never verified
[1784029357:191722][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 264704-265216 never verified
[1784029357:191726][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 276480-276992 never verified
[1784029357:191730][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 287744-288256 never verified
[1784029357:191734][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 290816-291328 never verified
[1784029357:191738][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 292352-292864 never verified
[1784029357:191742][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 346624-347136 never verified
[1784029357:191746][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 348160-348672 never verified
[1784029357:191750][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 355840-356352 never verified
[1784029357:191754][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 358400-358912 never verified
[1784029357:191758][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 361472-361984 never verified
[1784029357:191762][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 380416-380928 never verified
[1784029357:191766][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 381440-381952 never verified
[1784029357:191770][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 395264-408576 never verified
[1784029357:191775][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 425984-427008 never verified
[1784029357:191779][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 438784-439296 never verified
[1784029357:191784][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 439808-440832 never verified
[1784029357:191788][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 441344-443392 never verified
[1784029357:191792][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 443904-444416 never verified
[1784029357:191796][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 478720-479232 never verified
[1784029357:191800][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 481280-495104 never verified
[1784029357:191805][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 496128-496640 never verified
[1784029357:191817][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 506880-507392 never verified
[1784029357:191822][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 507904-508416 never verified
[1784029357:191826][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 508928-509952 never verified
[1784029357:191830][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 568320-568832 never verified
[1784029357:191841][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 574976-575488 never verified
[1784029357:191845][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 579584-580096 never verified
[1784029357:191849][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 583168-583680 never verified
[1784029357:191860][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 603648-604160 never verified
[1784029357:191864][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 606208-606720 never verified
[1784029357:191868][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 611840-612352 never verified
[1784029357:191879][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 614912-615424 never verified
[1784029357:191883][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 616960-617472 never verified
[1784029357:191887][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 617984-618496 never verified
[1784029357:191891][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 620544-621056 never verified
[1784029357:191902][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 635392-635904 never verified
[1784029357:191906][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 646144-646656 never verified
[1784029357:191910][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 666624-667136 never verified
[1784029357:191914][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 668672-669184 never verified
[1784029357:191919][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 669696-670208 never verified
[1784029357:191923][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 672768-673280 never verified
[1784029357:191927][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 676352-676864 never verified
[1784029357:191931][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 685568-686080 never verified
[1784029357:191936][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 687616-688128 never verified
[1784029357:191940][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 739328-739840 never verified
[1784029357:191945][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 742912-743424 never verified
[1784029357:191956][419936:0x7f6d2e7fc700], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 517: checkpoint range 797696-798208 never verified
[1784029357:19196

logs

format-test task_log

Logs:

t: FAILED: table_verify/62: ret == 0 || ret == EBUSY || (g.disagg_storage_config && !g.disagg_leader && ret == ENOENT)

logs

format-test task_log

Logs:

t: process aborting
WiredTiger Error: __wt_abort, 29: aborting WiredTiger library

logs

format-test task_log

Logs:

The following tests FAILED:
	  1 - test_format (Failed)

logs

format-test task_log

Logs:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f6d48f81859 in __GI_abort () at abort.c:79
#2  0x00007f6d493710cb in __wt_abort (session=session@entry=0x0) at /data/mci/6120bf198e798259601d6ac9ee1501e8/wiredtiger/src/os_common/os_abort.c:32
#3  0x0000000000423be0 in testutil_die (e=e@entry=0, fmt=fmt@entry=0x426020 "%s/%d: %s") at /data/mci/6120bf198e798259601d6ac9ee1501e8/wiredtiger/test/utility/misc.c:75
#4  0x000000000041c574 in table_verify (table=<optimized out>, arg=arg@entry=0x7f6d0c555860) at /data/mci/6120bf198e798259601d6ac9ee1501e8/wiredtiger/test/format/verify.c:62
#5  0x000000000041c6a4 in tables_apply (func=0x41c480 <table_verify>, arg=0x7f6d0c555860) at /data/mci/6120bf198e798259601d6ac9ee1501e8/wiredtiger/test/format/format_inline.h:186
#6  wts_verify (conn=0x7f6d0c555860, mirror_check=mirror_check@entry=true) at /data/mci/6120bf198e798259601d6ac9ee1501e8/wiredtiger/test/format/verify.c:398
#7  0x0000000000406ff2 in check_copy (session=0x1657968, id=<optimized out>) at /data/mci/6120bf198e798259601d6ac9ee1501e8/wiredtiger/test/format/backup.c:58
#8  backup (arg=<optimized out>) at /data/mci/6120bf198e798259601d6ac9ee1501e8/wiredtiger/test/format/backup.c:625
#9  0x00007f6d49159609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#10 0x00007f6d4907e353 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

logs

Repro Artifacts:

CONFIG

Assignee:: [DO NOT USE] Backlog - Storage Engines Team
Reporter:: xgen-buildbaron-user
Votes:: 0 Vote for this issue
Watchers:: 2 Start watching this issue

Created:: Jul 14 2026 11:49:00 PM UTC
Updated:: Jul 23 2026 11:24:11 PM UTC

Details

Description

Attachments

Activity

People

Dates