Add debugging for logging getting stuck

Verbose Error message

[1661231527:736274][38202:0x7f9a80ff9700], eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_server, 473: Cache stuck for too long, giving up: Connection timed out

Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_stress_tests_recovery_stress_test_1_42ee89096f8306dbdcbd1a9416c308cf2e0ccf7e_22_08_23_03_32_11/0?type=T#L7307

Cache Stuck Failures

transaction state dump

current ID: 56041

last running ID: 55650

metadata_pinned ID: 20506

oldest ID: 54407

durable timestamp: (0, 54980)

oldest timestamp: (0, 16915)

pinned timestamp: (0, 16915)

stable timestamp: (0, 16915)

has_durable_timestamp: yes

has_oldest_timestamp: yes

has_pinned_timestamp: yes

has_stable_timestamp: yes

oldest_is_pinned: yes

stable_is_pinned: yes

checkpoint running: yes

checkpoint generation: 2

checkpoint pinned ID: 0

checkpoint txn ID: 0

session count: 194

Transaction state of active sessions:

ID: 55650, pinned ID: 54407, metadata pinned ID: 20506, name: EMPTY

transaction id: 55650, mod count: 1, snap min: 20506, snap max: 55637, snapshot count: 147, commit_timestamp: (0, 54624), durable_timestamp: (0, 54624), first_commit_timestamp: (0, 54623), prepare_timestamp: (0, 0), pinned_durable_timestamp: (0, 54623), read_timestamp: (0, 0), checkpoint LSN: [0][0], full checkpoint: false, rollback reason: , flags: 0x0000301c, isolation: WT_ISO_SNAPSHOT

ID: 56002, pinned ID: 54407, metadata pinned ID: 0, name: EMPTY

transaction id: 56002, mod count: 1, snap min: 54407, snap max: 55994, snapshot count: 154, commit_timestamp: (0, 54964), durable_timestamp: (0, 54964), first_commit_timestamp: (0, 54963), prepare_timestamp: (0, 0), pinned_durable_timestamp: (0, 54963), read_timestamp: (0, 0), checkpoint LSN: [0][0], full checkpoint: false, rollback reason: , flags: 0x0000301c, isolation: WT_ISO_SNAPSHOT

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

cache dump

cache full: no

cache clean check: yes (95.010%)

cache dirty check: no (89.760%)

cache updates check: no (72.071%)

file:oplog.wt(<live>):

internal: 1 pages, 30.79 KB, 1/0 clean/dirty pages, 30.79/0.00 clean / dirty KB, 30.79 KB max page, 0.00 KB max dirty page

leaf: 174 pages, 15317.83 KB, 35/139 clean/dirty pages, 593.03 /14724.79 /11753.77 clean/dirty/updates KB, 206.28 KB max page, 206.28 KB max dirty page

file:local.wt(<live>):

internal: 1 pages, 33.43 KB, 1/0 clean/dirty pages, 33.43/0.00 clean / dirty KB, 33.43 KB max page, 0.00 KB max dirty page

leaf: 189 pages, 15806.64 KB, 46/143 clean/dirty pages, 815.72 /14990.92 /11776.90 clean/dirty/updates KB, 206.24 KB max page, 206.24 KB max dirty page

file:shadow.wt(<live>):

internal: 1 pages, 36.03 KB, 1/0 clean/dirty pages, 36.03/0.00 clean / dirty KB, 36.03 KB max page, 0.00 KB max dirty page

leaf: 204 pages, 15437.29 KB, 57/147 clean/dirty pages, 920.93 /14516.37 /11799.14 clean/dirty/updates KB, 208.92 KB max page, 208.92 KB max dirty page

file:collection.wt(<live>):

internal: 1 pages, 36.59 KB, 1/0 clean/dirty pages, 36.59/0.00 clean / dirty KB, 36.59 KB max page, 0.00 KB max dirty page

leaf: 207 pages, 15439.76 KB, 58/149 clean/dirty pages, 949.01 /14490.75 /11803.69 clean/dirty/updates KB, 203.53 KB max page, 203.53 KB max dirty page

file:WiredTigerHS.wt(<live>) eviction disabled at open:

internal: 1 pages, 0.27 KB, 1/0 clean/dirty pages, 0.27/0.00 clean / dirty KB, 0.27 KB max page, 0.00 KB max dirty page

leaf: 1 pages, 0.09 KB, 1/0 clean/dirty pages, 0.09 /0.00 /0.00 clean/dirty/updates KB, 0.09 KB max page, 0.00 KB max dirty page

file:WiredTiger.wt(<live>):

internal: 1 pages, 0.56 KB, 1/0 clean/dirty pages, 0.56/0.00 clean / dirty KB, 0.56 KB max page, 0.00 KB max dirty page

leaf: 1 pages, 16.82 KB, 1/0 clean/dirty pages, 16.82 /0.00 /16.73 clean/dirty/updates KB, 16.82 KB max page, 0.00 KB max dirty page

cache dump: total found: 65.56 MB vs tracked inuse 60.70 MB

total dirty bytes: 57.35 MB vs tracked dirty 57.35 MB

Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_stress_tests_recovery_stress_test_1_42ee89096f8306dbdcbd1a9416c308cf2e0ccf7e_22_08_23_03_32_11/0?type=T#L7423

Verbose Error message

[1661231527:737070][38202:0x7f9a80ff9700], eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_evict_thread_run, 324: cache eviction thread error: Connection timed out

[1661231527:737081][38202:0x7f9a80ff9700], eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_evict_thread_run, 324: the process must exit and restart: WT_PANIC: WiredTiger library panic

[1661231527:737091][38202:0x7f9a80ff9700], eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 28: aborting WiredTiger library

Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_stress_tests_recovery_stress_test_1_42ee89096f8306dbdcbd1a9416c308cf2e0ccf7e_22_08_23_03_32_11/0?type=T#L7475

C/CPP Suite Failure message

test_timestamp_abort: FAILED: Child process 38202 abnormally exited: Invalid argument

Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_stress_tests_recovery_stress_test_1_42ee89096f8306dbdcbd1a9416c308cf2e0ccf7e_22_08_23_03_32_11/0?type=T#L7481

Failure keywords

test_timestamp_abort: process aborting

WiredTiger Error: __wt_abort, 28: aborting WiredTiger library

Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_stress_tests_recovery_stress_test_1_42ee89096f8306dbdcbd1a9416c308cf2e0ccf7e_22_08_23_03_32_11/0?type=T#L7482

Evergreen YAML Script Failure

Command failed: shell script encountered problem: exit code 134

Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_stress_tests_recovery_stress_test_1_42ee89096f8306dbdcbd1a9416c308cf2e0ccf7e_22_08_23_03_32_11/0?type=T#L7485