[WT-9303] Fix checkpoint failure in verify operation - no space on device - MongoDB Jira

XML

Word

Printable

JSON

Details

Type: Build Failure
Status: Closed
Priority: Major - P3
Resolution: Works as Designed
Affects Version/s: None
Fix Version/s: None
Component/s: None
Labels:
- BB-Tools
- stability

Evergreen Project:
- wiredtiger
Failure Type:
- failed
Failing Buildvariants:
- ubuntu2004
Failing Tasks:
- spinlock-gcc-test
- spinlock-pthread-adaptive-test
First Failing Revision:
- a1d0eda743a51dd607743af19eef48773e650640
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-9303
Count of Linked BFGs (Last 30 days):
1

Story Points:
5
Sprint:
Storage Engines - 2022-06-27

Description

spinlock-pthread-adaptive-test on ubuntu2004

Host: i-0d1b10211b2505929
Project: wiredtiger
Commit: a1d0eda7

Task Logs:

task spinlock-pthread-adaptive-test

spinlock-pthread-adaptive-test task_log

Logs:

=======================================================

ERROR: test_alter02.test_alter02.test_alter02_log(s3.conn-always-logged.lsm.create-logged.reopen) (subunit.RemotedTestCase)

test_alter02.test_alter02.test_alter02_log(s3.conn-always-logged.lsm.create-logged.reopen)

----------------------------------------------------------------------

testtools.testresult.real._StringException: Traceback (most recent call last):

  File "/data/mci/45b522c9b6701535dde8c9578c525162/wiredtiger/test/suite/wttest.py", line 519, in setUp

    os.makedirs(self.testdir)

  File "/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/python3-v3.9MG/lib/python3.9/os.py", line 225, in makedirs

    mkdir(name, mode)

OSError: [Errno 28] No space left on device: 'WT_TEST/test_alter02.test_alter02.test_alter02_log_s3.conn-always-logged.lsm.create-logged.reopen'

======================================================================

ERROR: test_a

Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_spinlock_pthread_adaptive_test_a1d0eda743a51dd607743af19eef48773e650640_22_05_09_15_39_55/0?type=T#L20160

Verbose Error message

[1652117637:605608][53148:0x7fc126818740], test_alter04.test_alter04.test_alter04_cache(non_tiered.table-cg.200K.reopen.cache), file:WiredTiger.wt, WT_CONNECTION.close: [WT_VERB_DEFAULT][ERROR]: __wt_block_checkpoint_resolve, 928: the process must exit and restart: WT_PANIC: WiredTiger library panic

[1652117637:605669][53148:0x7fc126818740], test_alter04.test_alter04.test_alter04_cache(non_tiered.table-cg.200K.reopen.cache), file:WiredTiger.wt, WT_CONNECTION.close: [WT_VERB_DEFAULT][ERROR]: __wt_conn_dhandle_discard_single, 871: Final close of file:WiredTiger.wt failed: WT_PANIC: WiredTiger library panic

Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_spinlock_pthread_adaptive_test_a1d0eda743a51dd607743af19eef48773e650640_22_05_09_15_39_55/0?type=T#L102920

Evergreen YAML Script Failure

Command failed: shell script encountered problem: exit code 1

Command failed: shell script encountered problem: exit code 1

Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_spinlock_pthread_adaptive_test_a1d0eda743a51dd607743af19eef48773e650640_22_05_09_15_39_55/0?type=T#L102964

Verbose Error message

[1652117637:604700][53154:0x7fc1222ca700], log-server: [WT_VERB_DEFAULT][ERROR]: __posix_open_file, 801: ././WiredTigerTmplog.0000000002: handle-open: open: No space left on device

[1652117637:607217][53154:0x7fc1222ca700], log-server: [WT_VERB_DEFAULT][ERROR]: __log_prealloc_once, 510: log pre-alloc server error: No space left on device

[1652117637:607254][53154:0x7fc1222ca700], log-server: [WT_VERB_DEFAULT][ERROR]: __log_server, 914: log server error: No space left on device

[1652117637:607275][53154:0x7fc1222ca700], log-server: [WT_VERB_DEFAULT][ERROR]: __log_server, 914: the process must exit and restart: WT_PANIC: WiredTiger library panic

[1652117637:608163][53154:0x7fc126818740], file:WiredTiger.wt, close_ckpt: [WT_VERB_DEFAULT][ERROR]: __posix_open_file, 801: ./WiredTiger.turtle.set: handle-open: open: No space left on device

[1652117637:608205][53154:0x7fc126818740], file:WiredTiger.wt, close_ckpt: [WT_VERB_DEFAULT][ERROR]: __wt_block_checkpoint_resolve, 928: WiredTiger.wt: the checkpoint failed, the system must restart: Invalid argument

[1652117637:608229][53154:0x7fc126818740], close_ckpt: [WT_VERB_DEFAULT][ERROR]: __wt_cursor_set_key, 289: failed to set key: WT_PANIC: WiredTiger library panic

[1652117637:608243][53154:0x7fc126818740], close_ckpt: [WT_VERB_DEFAULT][ERROR]: __meta_track_unroll, 213: metadata unroll update system:checkpoint_base_write_gen to base_write_gen=16: WT_PANIC: WiredTiger library panic

[1652117637:608257][53154:0x7fc126818740], close_ckpt: [WT_VERB_DEFAULT][ERROR]: __wt_cursor_set_key, 289: failed to set key: WT_PANIC: WiredTiger library panic

[1652117637:608268][53154:0x7fc126818740], close_ckpt: [WT_VERB_DEFAULT][ERROR]: __meta_track_unroll, 213: metadata unroll update system:checkpoint_snapshot to snapshot_min=1,snapshot_max=1,snapshot_count=0,checkpoint_time=1652117637,write_gen=16: WT_PANIC: WiredTiger library panic

[1652117637:608282][53154:0x7fc126818740], close_ckpt: [WT_VERB_DEFAULT][ERROR]: __wt_meta_track_off, 334: failed to apply or unroll all tracked operations: WT_PANIC: WiredTiger library panic

[1652117637:608378][53154:0x7fc126818740], WT_CONNECTION.close: [WT_VERB_DEFAULT][ERROR]: __conn_close, 1177: failure during close, disabling further writes: WT_PANIC: WiredTiger library panic

[1652117637:608458][53154:0x7fc11eac3700], eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_evict_thread_run, 324: cache eviction thread error: WT_PANIC: WiredTiger library panic

[1652117637:608506][53154:0x7fc121ac9700], eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_evict_thread_run, 324: cache eviction thread error: WT_PANIC: WiredTiger library panic

[1652117637:608556][53154:0x7fc1202c6700], eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_evict_thread_run, 324: cache eviction thread error: WT_PANIC: WiredTiger library panic

[1652117637:608627][53154:0x7fc1212c8700], eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_evict_thread_run, 324: cache eviction thread error: WT_PANIC: WiredTiger library panic

[1652117637:608661][53154:0x7fc11f2c4700], eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_evict_thread_run, 324: cache eviction thread error: WT_PANIC: WiredTiger library panic

[1652117637:608526][53154:0x7fc11fac5700], eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_evict_thread_run, 324: cache eviction thread error: WT_PANIC: WiredTiger library panic

[1652117637:608595][53154:0x7fc120ac7700], eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_evict_thread_run, 324: cache eviction thread error: WT_PANIC: WiredTiger library panic

[1652117637:609358][53154:0x7fc1242ce700], log-wrlsn-server: [WT_VERB_DEFAULT][ERROR]: __wt_log_slot_switch, 416: log slot switch fatal error: WT_PANIC: WiredTiger library panic

[1652117637:609588][53154:0x7fc1242ce700], log-wrlsn-server: [WT_VERB_DEFAULT][ERROR]: __log_wrlsn_server, 824: log wrlsn server error: WT_PANIC: WiredTiger library panic

[1652117637:609716][53154:0x7fc126818740], WT_CONNECTION.close: [WT_VERB_DEFAULT][ERROR]: __wt_cache_destroy, 364: cache server: exiting with 4 pages in memory and 0 pages evicted

[1652117637:604507][53148:0x7fc126818740], test_alter04.test_alter04.test_alter04_cache(non_tiered.table-cg.200K.reopen.cache), file:WiredTiger.wt, WT_CONNECTION.close: [WT_VERB_DEFAULT][ERROR]: __posix_open_file, 801: ./WiredTiger.turtle.set: handle-open: open: No space left on device

[1652117637:605576][53148:0x7fc126818740], test_alter04.test_alter04.test_alter04_cache(non_tiered.table-cg.200K.reopen.cache), file:WiredTiger.wt, WT_CONNECTION.close: [WT_VERB_DEFAULT][ERROR]: __wt_block_checkpoint_resolve, 928: WiredTiger.wt: the checkpoint failed, the system must restart: Invalid argument

[1652117637:605608][53148:0x7fc126818740], test_alter04.test_alter04.test_alter04_cache(non_tiered.table-cg.200K.reopen.cache), file:WiredTiger.wt, WT_CONNECTION.close: [WT_VERB_DEFAULT][ERROR]: __wt_block_checkpoint_resolve, 928: the process must exit and restart: WT_PANIC: WiredTiger library panic

[1652117637:605669][53148:0x7fc126818740], test_alter04.test_alter04.test_alter04_cache(non_tiered.table-cg.200K.reopen.cache), file:WiredTiger.wt, WT_CONNECTION.close: [WT_VERB_DEFAULT][ERROR]: __wt_conn_dhandle_discard_single, 871: Final close of file:WiredTiger.wt failed: WT_PANIC: WiredTiger library panic

[1652117637:602880][53150:0x7fc126818740], file:WiredTiger.wt, WT_CONNECTION.close: [WT_VERB_DEFAULT][ERROR]: __posix_open_file, 801: ./WiredTiger.turtle.set: handle-open: open: No space left on device

[1652117637:602969][53150:0x7fc126818740], file:WiredTiger.wt, WT_CONNECTION.close: [WT_VERB_DEFAULT][ERROR]: __wt_block_checkpoint_resolve, 928: WiredTiger.wt: the checkpoint failed, the system must restart: Invalid argument

[1652117637:602988][53150:0x7fc126818740], file:WiredTiger.wt, WT_CONNECTION.close: [WT_VERB_DEFAULT][ERROR]: __wt_block_checkpoint_resolve, 928: the process must exit and restart: WT_PANIC: WiredTiger library panic

[1652117637:603054][53150:0x7fc126818740], file:WiredTiger.wt, WT_CONNECTION.close: [WT_VERB_DEFAULT][ERROR]: __wt_conn_dhandle_discard_single, 871: Final close of file:WiredTiger.wt failed: WT_PANIC: WiredTiger library panic

[1652117637:603307][53150:0x7fc1222ca700], log-wrlsn-server: [WT_VERB_DEFAULT][ERROR]: __log_wrlsn_server, 824: log wrlsn server error: WT_PANIC: WiredTiger library panic

[1652117637:601947][53156:0x7fc126818740], connection: [WT_VERB_DEFAULT][ERROR]: __posix_open_file, 801: ./WiredTiger.wt: handle-open: open: No space left on device

[1652117637:603384][53156:0x7fc126818740], file:WiredTiger.wt, connection: [WT_VERB_DEFAULT][ERROR]: __posix_open_file, 801: ./WiredTiger.wt: handle-open: open: No such file or directory

Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_spinlock_pthread_adaptive_test_a1d0eda743a51dd607743af19eef48773e650640_22_05_09_15_39_55/0?type=T#L103020

Evergreen YAML Script Failure

Command failed: error opening target archive file /data/mci/45b522c9b6701535dde8c9578c525162/wiredtiger.tgz: open /data/mci/45b522c9b6701535dde8c9578c525162/wiredtiger.tgz: no space left on device

Command failed: missing file '/data/mci/45b522c9b6701535dde8c9578c525162/wiredtiger.tgz': problem opening file /data/mci/45b522c9b6701535dde8c9578c525162/wiredtiger.tgz: open /data/mci/45b522c9b6701535dde8c9578c525162/wiredtiger.tgz: no such file or directory

Command failed: error opening target archive file /data/mci/45b522c9b6701535dde8c9578c525162/wt-hang-analyzer.tgz: open /data/mci/45b522c9b6701535dde8c9578c525162/wt-hang-analyzer.tgz: no space left on device

Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_spinlock_pthread_adaptive_test_a1d0eda743a51dd607743af19eef48773e650640_22_05_09_15_39_55/0?type=T#L103078

logs

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

CONFIG.nospace
13 kB
May 31 2022 01:17:56 AM UTC

Activity

People

Assignee:: Keith Smith

Reporter:: xgen-buildbaron-user

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: May 10 2022 01:12:12 AM UTC

Updated:: Aug 17 2022 01:43:52 PM UTC

Resolved:: May 31 2022 01:31:48 AM UTC