-
Type:
Build Failure
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
Storage Engines - Foundations
-
229.131
-
None
-
None
-
3
format-stress-test-disagg-leader-data-validation-1 on amazon2023-arm64-nonstandalone
Host: i-069a8a8c283062e19
Project: wiredtiger
Commit: 6a0175cf
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules
Task Logs:
format-stress-test-disagg-leader-data-validation-1 task_log
Logs:
/opt/mongodbtoolchain/revisions/93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542/scripts/products.sh: line 13: /opt/mongodbtoolchain/revisions/93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542/scripts/installer.sh: No such file or directory
format-stress-test-disagg-leader-data-validation-1 task_log
Logs:
[1781853246:511932][33626:0xffff9098dbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_server, 278: Cache stuck for too long, giving up: Connection timed out
format-stress-test-disagg-leader-data-validation-1 task_log
Logs:
0x5204ffc97000:transaction state dump 0x5204ffc97000:current ID: 11895434 0x5204ffc97000:last running ID: 11895434 0x5204ffc97000:metadata_pinned ID: 11895434 0x5204ffc97000:oldest ID: 11895434 0x5204ffc97000:durable timestamp: (0, 31957351) 0x5204ffc97000:oldest timestamp: (0, 30937802) 0x5204ffc97000:pinned timestamp: (0, 30937802) 0x5204ffc97000:stable timestamp: (0, 30937802) 0x5204ffc97000:stable disaggregated schema epoch: (0, 0) 0x5204ffc97000:has_durable_timestamp: yes 0x5204ffc97000:has_oldest_timestamp: yes 0x5204ffc97000:has_pinned_timestamp: yes 0x5204ffc97000:has_stable_timestamp: yes 0x5204ffc97000:has_stable_disaggregated_schema_epoch: no 0x5204ffc97000:oldest_is_pinned: yes 0x5204ffc97000:stable_is_pinned: yes 0x5204ffc97000:checkpoint running: no 0x5204ffc97000:checkpoint generation: 22 0x5204ffc97000:checkpoint pinned ID: 0 0x5204ffc97000:checkpoint txn ID: 0 0x5204ffc97000:session count: 22 0x5204ffc97000:Transaction state of active sessions: 0x5204ffc97000:session ID: 21, txn ID: 0, pinned ID: 11895434, metadata pinned ID: 0, name: WT_CURSOR.search 0x5204ffc97000:transaction id: 0, mod count: 0, snap min: 11895434, snap max: 11895434, snapshot count: 0, snapshot: [], commit_timestamp: (0, 0), durable_timestamp: (0, 0), first_commit_timestamp: (0, 0), prepare_timestamp: (0, 0), prepared id: 0, pinned_durable_timestamp: (0, 0), read_timestamp: (0, 30937802), checkpoint LSN: [0,0], full checkpoint: false, flags: 0x00000a04, isolation: WT_ISO_SNAPSHOT, last saved error code: 0, last saved sub-level error code: -32000, last saved error message: last API call was successful 0x5204ffc97000:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= 0x5204ffc97000:cache dump 0x5204ffc97000:cache full: no 0x5204ffc97000:cache clean check: no (27.019%) 0x5204ffc97000:cache dirty check: yes (20.924%) 0x5204ffc97000:cache updates check: yes (10.488%) 0x5204ffc97000:file:T00003.wt_stable(<live>): 0x5204ffc97000:internal: 519 pages, 36221.37 KB, 5/514 clean/dirty pages, 261.98/35959.39 clean / dirty KB, 119.43 KB max page, 119.43 KB max dirty page 0x5204ffc97000:leaf: 184502 pages, 319792.38 KB, 83869/100633 clean/dirty pages, 113222.75 /206569.64 /116716.63 clean/dirty/updates KB, 2493.89 KB max page, 2493.89 KB max dirty page 0x5204ffc97000:file:T00002.wt_stable(<live>): 0x5204ffc97000:internal: 11 pages, 632.59 KB, 1/10 clean/dirty pages, 82.55/550.05 clean / dirty KB, 95.25 KB max page, 95.25 KB max dirty page 0x5204ffc97000:leaf: 2870 pages, 229327.13 KB, 130/2740 clean/dirty pages, 21112.37 /208214.76 /89541.18 clean/dirty/updates KB, 2493.89 KB max page, 2493.89 KB max dirty page 0x5204ffc97000:file:T00001.wt(<live>): 0x5204ffc97000:internal: 75 pages, 6065.75 KB, 8/67 clean/dirty pages, 2258.38/3807.37 clean / dirty KB, 482.78 KB max page, 177.33 KB max dirty page 0x5204ffc97000:leaf: 2240 pages, 194925.92 KB, 14/2226 clean/dirty pages, 246.77 /194679.15 /99225.23 clean/dirty/updates KB, 2397.38 KB max page, 2397.38 KB max dirty page 0x5204ffc97000:file:WiredTigerShared.wt_stable(<live>): 0x5204ffc97000:internal: 1 pages, 0.85 KB, 1/0 clean/dirty pages, 0.85/0.00 clean / dirty KB, 0.85 KB max page, 0.00 KB max dirty page 0x5204ffc97000:leaf: 0 pages 0x5204ffc97000:file:WiredTigerSharedHS.wt_stable(<live>): 0x5204ffc97000:internal: 1 pages, 0.79 KB, 1/0 clean/dirty pages, 0.79/0.00 clean / dirty KB, 0.79 KB max page, 0.00 KB max dirty page 0x5204ffc97000:leaf: 0 pages 0x5204ffc97000:file:WiredTigerSharedHS.wt_stable/WiredTigerCheckpoint.24(<live>) eviction disabled at open: 0x5204ffc97000:internal: 1 pages, 0.48 KB, 1/0 clean/dirty pages, 0.48/0.00 clean / dirty KB, 0.48 KB max page, 0.00 KB max dirty page 0x5204ffc97000:leaf: 0 pages 0x5204ffc97000:file:WiredTigerHS.wt(<live>): 0x5204ffc97000:internal: 1 pages, 0.76 KB, 0/1 clean/dirty pages, 0.00/0.76 clean / dirty KB, 0.76 KB max page, 0.76 KB max dirty page 0x5204ffc97000:leaf: 0 pages 0x5204ffc97000:file:WiredTiger.wt(<live>): 0x5204ffc97000:internal: 1 pages, 0.77 KB, 1/0 clean/dirty pages, 0.77/0.00 clean / dirty KB, 0.77 KB max page, 0.00 KB max dirty page 0x5204ffc97000:leaf: 1 pages, 14.07 KB, 1/0 clean/dirty pages, 14.07 /0.00 /0.00 clean/dirty/updates KB, 14.07 KB max page, 0.00 KB max dirty page 0x5204ffc97000:cache dump: total found: 830.02 MB vs tracked inuse 768.54 MB 0x5204ffc97000:total dirty bytes: 634.55 MB vs tracked dirty 634.55 MB
format-stress-test-disagg-leader-data-validation-1 task_log
Logs:
[1781853246:577703][33626:0xffff9098dbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_btcur_next, 795: Error at src/btree/bt_curnext.c:795: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found [1781853246:577713][33626:0xffff9098dbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_next, 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found [1781853246:577716][33626:0xffff9098dbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_thread_run, 98: Error at src/evict/evict_thread.c:98: "ret" failed: Connection timed out [1781853246:577720][33626:0xffff9098dbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 121: eviction thread error: Connection timed out [1781853246:577722][33626:0xffff9098dbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 121: the process must exit and restart: WT_PANIC: WiredTiger library panic [1781853246:577726][33626:0xffff9098dbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library
format-stress-test-disagg-leader-data-validation-1 task_log
Logs:
#0 0x0000ffff94ac4454 in __pthread_kill_implementation () from /lib64/libc.so.6 #0 0x0000ffff94ac4454 in __pthread_kill_implementation () from /lib64/libc.so.6 #1 0x0000ffff94a7b320 [PAC] in raise () from /lib64/libc.so.6 #2 0x0000ffff94a62224 [PAC] in abort () from /lib64/libc.so.6 #3 0x0000ffff94e54f38 [PAC] in __wt_abort (session=session@entry=0x5204ffc97000) at /data/mci/0b451b5d33018fa3f1ae6702d17741c1/wiredtiger/src/os_common/os_abort.c:32 #4 0x0000ffff94f005c4 in __wt_panic_func (session=session@entry=0x5204ffc97000, error=error@entry=110, func=func@entry=0xffff95006380 <__PRETTY_FUNCTION__.6> "__evict_thread_run", line=line@entry=121, category=category@entry=WT_VERB_DEFAULT, fmt=fmt@entry=0xffff94fa0af0 "eviction thread error") at /data/mci/0b451b5d33018fa3f1ae6702d17741c1/wiredtiger/src/support/err.c:633 #5 0x0000ffff94e0d328 in __evict_thread_run (session=0x5204ffc97000, thread=0x5204ffe2f1d0) at /data/mci/0b451b5d33018fa3f1ae6702d17741c1/wiredtiger/src/evict/evict_thread.c:121 #6 0x0000ffff94f1c508 in __thread_run (arg=0x5204ffe2f1d0) at /data/mci/0b451b5d33018fa3f1ae6702d17741c1/wiredtiger/src/support/thread_group.c:32 #7 0x0000ffff94ac2834 in start_thread () from /lib64/libc.so.6 #8 0x0000ffff94a66e5c [PAC] in thread_start () from /lib64/libc.so.6
Repro Artifacts: