Loading...

XML

Word

Printable

JSON

Type: Bug
Resolution: Fixed
Priority: Critical - P2
Fix Version/s: WT12.0.0, 8.3.0-rc0
Affects Version/s: None
Component/s: Checkpoints
Labels:
- Disag_Storage
- dc

Assigned Teams:

Storage Engines, Storage Engines - Transactions
Sprint:
SE Transactions - 2025-08-29, SE Transactions - 2025-09-12
Story Points:
5
Evergreen Project:
- wiredtiger
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-15299
Count of Linked BFGs (Last 30 days):
3

We hit a data mismatch in recovery test with precise checkpoint:

[2025/08/27 16:11:00.803] WT open config: create,cache_cursors=false,statistics=(all),statistics_log=(json,on_close,wait=1),log=(enabled),error_prefix="test_checkpoint",cache_size=1G, eviction_dirty_trigger=5, eviction_dirty_target=1,,debug_mode=(eviction=true,table_logging=true),verbose=(recovery)
[2025/08/27 16:11:00.803] [1756275060:418840][26738:0x7f57c11c9300], test_checkpoint, wiredtiger_open: [WT_VERB_RECOVERY][INFO]: opening the WiredTiger library
[2025/08/27 16:11:00.803] [1756275060:420364][26738:0x7f57c11c9300], test_checkpoint, connection: [WT_VERB_RECOVERY][INFO]: connection configuration string parsing completed
[2025/08/27 16:11:00.803] [1756275060:422035][26738:0x7f57c11c9300], test_checkpoint, connection: [WT_VERB_RECOVERY][INFO]: starting WiredTiger utility threads
[2025/08/27 16:11:00.803] [1756275060:433553][26738:0x7f57c11c9300], test_checkpoint, connection: [WT_VERB_RECOVERY][INFO]: starting WiredTiger recovery
[2025/08/27 16:11:00.803] [1756275060:433645][26738:0x7f57c11c9300], test_checkpoint, txn-recover: [WT_VERB_RECOVERY][DEBUG_1]: Recovering file:WiredTiger.wt with id 0 @ (1,16910976)
[2025/08/27 16:11:00.803] [1756275060:555236][26738:0x7f57c11c9300], test_checkpoint, txn-recover: [WT_VERB_RECOVERY][INFO]: scanning metadata to find the largest file ID
[2025/08/27 16:11:00.803] [1756275060:555326][26738:0x7f57c11c9300], test_checkpoint, txn-recover: [WT_VERB_RECOVERY][DEBUG_1]: Recovering file:WiredTigerHS.wt with id 8 @ (1,16910976)
[2025/08/27 16:11:00.803] [1756275060:555392][26738:0x7f57c11c9300], test_checkpoint, txn-recover: [WT_VERB_RECOVERY][DEBUG_1]: Recovering file:__wt0000.wt with id 16 @ (1,16910976)
[2025/08/27 16:11:00.803] [1756275060:555443][26738:0x7f57c11c9300], test_checkpoint, txn-recover: [WT_VERB_RECOVERY][DEBUG_1]: Recovering file:__wt0001.wt with id 24 @ (1,16910976)
[2025/08/27 16:11:00.803] [1756275060:555490][26738:0x7f57c11c9300], test_checkpoint, txn-recover: [WT_VERB_RECOVERY][DEBUG_1]: Recovering file:__wt0002.wt with id 32 @ (1,16910976)
[2025/08/27 16:11:00.803] [1756275060:555528][26738:0x7f57c11c9300], test_checkpoint, txn-recover: [WT_VERB_RECOVERY][INFO]: largest file ID found in the metadata 32
[2025/08/27 16:11:00.803] [1756275060:562645][26738:0x7f57c11c9300], test_checkpoint, txn-recover: [WT_VERB_RECOVERY][INFO]: Main recovery loop: starting at 1,16910976 to 2,256
[2025/08/27 16:11:00.803] [1756275060:689964][26738:0x7f57c11c9300], test_checkpoint, txn-recover: [WT_VERB_RECOVERY][INFO]: recovery log replay has successfully finished and ran for 256 milliseconds
[2025/08/27 16:11:00.803] [1756275060:690027][26738:0x7f57c11c9300], test_checkpoint, txn-recover: [WT_VERB_RECOVERY][DEBUG_1]: Recovery checkpoint_timestamp 15be
[2025/08/27 16:11:00.803] [1756275060:690038][26738:0x7f57c11c9300], test_checkpoint, txn-recover: [WT_VERB_RECOVERY][DEBUG_1]: Set global recovery timestamp: (0, 5566)
[2025/08/27 16:11:00.803] [1756275060:690047][26738:0x7f57c11c9300], test_checkpoint, txn-recover: [WT_VERB_RECOVERY][DEBUG_1]: Recovery oldest_timestamp baf
[2025/08/27 16:11:00.803] [1756275060:690052][26738:0x7f57c11c9300], test_checkpoint, txn-recover: [WT_VERB_RECOVERY][DEBUG_1]: Set global oldest timestamp: (0, 2991)
[2025/08/27 16:11:00.803] [1756275060:696596][26738:0x7f57c11c9300], test_checkpoint, txn-recover: [WT_VERB_RECOVERY][INFO]: recovery checkpoint has successfully finished and ran for 5 milliseconds
[2025/08/27 16:11:00.803] [1756275060:696642][26738:0x7f57c11c9300], test_checkpoint, txn-recover: [WT_VERB_RECOVERY][INFO]: recovery was completed successfully and took 263ms, including 256ms for the log replay, 0ms for the rollback to stable, and 5ms for the checkpoint.
[2025/08/27 16:11:00.803] [1756275060:699065][26738:0x7f57c11c9300], test_checkpoint, connection: [WT_VERB_RECOVERY][INFO]: WiredTiger utility threads started successfully
[2025/08/27 16:11:00.803] [1756275060:699080][26738:0x7f57c11c9300], test_checkpoint, connection: [WT_VERB_RECOVERY][INFO]: the WiredTiger library has successfully opened
[2025/08/27 16:11:00.803] Key/value mismatch: 6737/0000000000000000000000000000000000000000000000077788 (192) from a ROW table is not 6736/0000000000000000000000000000000000000000000000017993 (196) from a ROW table
[2025/08/27 16:11:00.803] [1756275060:769285][26738:0x7f57c11c9300], test_checkpoint, WT_SESSION.open_cursor: [WT_VERB_RECOVERY][DEBUG_1]: Recovery checkpoint_timestamp 15be
[2025/08/27 16:11:00.803] [1756275060:769300][26738:0x7f57c11c9300], test_checkpoint, WT_SESSION.open_cursor: [WT_VERB_RECOVERY][DEBUG_1]: Recovery oldest_timestamp baf
[2025/08/27 16:11:00.803] test_checkpoint: diagnose_key_error,698: 1st cursor didn't find 2nd key: WT_NOTFOUND: item not found
[2025/08/27 16:11:00.803] [1756275060:772283][26738:0x7f57c11c9300], test_checkpoint, WT_SESSION.open_cursor: [WT_VERB_RECOVERY][DEBUG_1]: Recovery checkpoint_timestamp 15be
[2025/08/27 16:11:00.803] [1756275060:772296][26738:0x7f57c11c9300], test_checkpoint, WT_SESSION.open_cursor: [WT_VERB_RECOVERY][DEBUG_1]: Recovery oldest_timestamp baf
[2025/08/27 16:11:00.803] test_checkpoint: verify_consistency,524: verify_consistency - mismatching data: Bad address

causes

WT-15335 failed: unit-test-bucket03 on ubuntu2004 [wiredtiger @ d366e912]

Closed

is duplicated by

WT-15278 failed: data-validation-stress-test-precise-checkpoint-fp-hs-insert-s2 on ubuntu2004-nonstandalone [wiredtiger @ 13a54b49]

Closed

is related to

WT-15309 Memory leak when allocating multi->block_meta

Closed

WT-15373 Fix a memory leak in __disagg_pick_up_checkpoint

Closed

WT-15314 Introduce WT disagg waterfall testing to auto triage disagg tests

Closed

WT-15316 Update nlohmann/json to avoid compilation warnings

Closed

WT-15330 Remove claimed prepared transactions from hash map

Closed

related to

WT-15214 test/format (disagg) hits WT_PANIC on file:wt.wt_stable, unencrypted block with encryption configured

Closed

WT-15238 Found an update with max stop timestamp when inserting data to the history store

Closed

WT-14931 Add preserve prepare config tests to test format and test checkpoint

Closed

WT-15157 Disagg testing: add leader mode to test/checkpoint

Closed

WT-15165 Design a data structure to temporarily store the identified prepared transactions in recovery

Closed

WT-15314 Introduce WT disagg waterfall testing to auto triage disagg tests

Closed

WT-15011 Return more specific error message about what caused WT_WRITE_CONFLICT

Closed

WT-15008 make-check-test times out on windows due to slow machines

Closed

(2 is related to, 8 related to)

Assignee:: Chenhao Qu
Reporter:: Chenhao Qu
Votes:: 0 Vote for this issue
Watchers:: 3 Start watching this issue

Created:: Aug 28 2025 12:03:17 AM UTC
Updated:: Sep 11 2025 05:35:58 AM UTC
Resolved:: Sep 08 2025 02:46:32 AM UTC

Details

Description

Attachments

Issue Links

Activity

People

Dates