Loading...

XML

Word

Printable

JSON

Type: Build Failure
Resolution: Duplicate
Priority: Minor - P4
Fix Version/s: None
Affects Version/s: None
Component/s: None
Labels:

Sprint:
NachoCheese - 2023-10-03
Story Points:
None
Evergreen Project:
Failure Type:
- failed
Failing Buildvariants:
- rhel80
- ubuntu2004
Failing Tasks:
- spinlock-gcc-test
- spinlock-pthread-adaptive-test
First Failing Revision:
- fc5838e9095b279533f151b03dc31ff2f3756b00
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-9843
Count of Linked BFGs (Last 30 days):
0

This test failed part-way through the python test suite with an out of space error. All of the remaining python tests at that point failed for the same reason. The goal on this ticket should be to understand why this particular test run ran out of disk space and, if needed, adjust our evergreen testing to make this less likely.

spinlock-pthread-adaptive-test on ubuntu2004

Host: i-0ea55d06dfca15d54
Project: wiredtiger
Commit: fc5838e9

Task Logs:

task spinlock-pthread-adaptive-test

spinlock-pthread-adaptive-test task_log

Logs:

t connection during test 'test_checkpoint_snapshot02.test_checkpoint_snapshot02.test_checkpoint_snapshot_with_txnid_and_timestamp(column_fix.backup)'
======================================================================
ERROR: test_compat01.test_compat01.test_restart(old2.old_patch) (subunit.RemotedTestCase)
test_compat01.test_compat01.test_restart(old2.old_patch)
----------------------------------------------------------------------
testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger/test/suite/wttest.py", line 522, in setUp
    os.makedirs(self.testdir)
  File "/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/python3-v3.9MG/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
OSError: [Errno 28] No space left on device: 'WT_TEST/test_compat01.test_compat01.test_restart_old2.old_patch'
======================================================================
ERROR: test_compat01.test_compat01.test_restart(old_patch2.30_patch) (subunit.RemotedTestCase)
test_compat01.test_compat01.test_restart(old_patch2.30_patch)
----------------------------------------------------------------------
testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger/test/suite/wttest.py", line 522, in setUp
    os.makedirs(self.testdir)
  File "/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/python3-v3.9MG/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
OSError: [Errno 28] No space left on device: 'WT_TEST/test_compat01.test_compat01.test_restart_old_patch2.30_patch'
======================================================================
ERROR: test_compat02.test_compat02.test_compat02(def.100_rel.def_min.110_max.basecfg_false) (subunit.RemotedTestCase)
test_compat02.test_compat02.test_compat02(def.100_rel.def_min.110_max.basecfg_false)
----------------------------------------------------------------------
testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger/test/suite/wttest.py", line 522, in setUp
    os.makedirs(self.testdir)
  File "/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/python3-v3.9MG/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
OSError: [Errno 28] No space left on device: 'WT_TEST/test_compat02.test_compat02.test_compat02_def.100_rel.def_min.110_max.basecfg_false'
======================================================================
ERROR: test_compat02.test_compat02.test_compat02(def.26_rel.100_min.def_max.basecfg_false) (subunit.RemotedTestCase)
test_compat02.test_compat02.test_compat02(def.26_rel.100_min.def_max.basecfg_false)
----------------------------------------------------------------------
testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger/test/suite/wttest.py", line 522, in setUp
    os.makedirs(self.testdir)
  File "/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/python3-v3.9MG/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
OSError: [Errno 28] No space left on device: 'WT_TEST/test_compat02.test_compat02.test_compat02_def.26_rel.100_min.def_max.basecfg_false'
======================================================================
ERROR: test_compat02.test_compat02.test_compat02(110.31_rel.33_min.33_max.basecfg_false) (subunit.RemotedTestCase)
test_compat02.test_compat02.test_compat02(110.31_rel.33_min.33_max.basecfg_false)
----------------------------------------------------------------------
testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger/test/suite/wttest.py", line 522, in setUp
    os.makedirs(self.testdir)
  File "/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/python3-v3.9MG/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
OSError: [Errno 28] No space left on device: 'WT_TEST/test_compat02.test_compat02.test_compat02_110.31_rel.33_min.33_max.basecfg_false'
======================================================================
ERROR: test_compat02.test_compat02.test_compat02(110.26_rel.30_min.future_max.basecfg_true) (subunit.RemotedTestCase)
test_compat02.test_compat02.test_compat02(110.26_rel.30_min.future_max.basecfg_true)
----------------------------------------------------------------------
testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger/test/suite/wttest.py", line 522, in setUp
    os.makedirs(self.testdir)
  File "/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/python3-v3.9MG/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
OSError: [Errno 28] No space left on device: 'WT_TEST/test_compat02.test_compat02.test_compat02_110.26_rel.30_min.future_max.basecfg_true'
======================================================================
ERROR: test_compat02.test_compat02.test_compat02(100.30_rel.future_min.32_max.basecfg_false) (subunit.RemotedTestCase)
test_compat02.test_compat02.test_compat02(100.30_rel.future_min.32_max.basecfg_false)
----------------------------------------------------------------------
testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger/test/suite/wttest.py", line 522, in setUp
    os.makedirs(self.testdir)
  File "/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/python3-v3.9MG/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
OSError: [Errno 28] No space left on device: 'WT_TEST/test_compat02.test_compat02.test_compat02_100.30_rel.future_min.32_max.basecfg_false'
======================================================================
ERROR: test_compat02.test_compat02.test_compat02(33.100_rel.26_min.30_max.basecfg_false) (subunit.RemotedTestCase)
test_compat02.test_compat02.test_compat02(33.100_rel.26_min.30_max.basecfg_false)
----------------------------------------------------------------------
testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger/test/suite/wttest.py", line 522, in setUp
    os.makedirs(self.testdir)
  File "/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/python3-v3.9MG/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
OSError: [Errno 28] No space left on device: 'WT_TEST/test_compat02.test_compat02.test_compat02_33.100_rel.26_min.30_max.basecfg_false'
======================================================================
ERROR: test_compat02.test_compat02.test_compat02(32.110_rel.future_min.def_max.basecfg_true) (subunit.RemotedTestCase)
test_compat02.test_compat02.test_compat02(32.110_rel.future_min.def_max.basecfg_true)
----------------------------------------------------------------------
testtools.testresul
Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_spinlock_pthread_adaptive_test_fc5838e9095b279533f151b03dc31ff2f3756b00_22_08_31_18_09_21/0?type=T#L8213

Evergreen YAML Script Failure
Command failed: shell script encountered problem: exit code 1

Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_spinlock_pthread_adaptive_test_fc5838e9095b279533f151b03dc31ff2f3756b00_22_08_31_18_09_21/0?type=T#L19382

Verbose Error message
[1661976154:254767][44383:0x7f3f5260e700], test_checkpoint_snapshot02.test_checkpoint_snapshot02.test_checkpoint_snapshot_with_txnid_and_timestamp(row_integer.crash_restart), statlog-server: [WT_VERB_DEFAULT][ERROR]: __posix_file_write, 611: BACKUP/./WiredTigerStat.31.20: handle-write: pwrite: failed to write 7136 bytes at offset 663552: No space left on device
[1661976154:254852][44383:0x7f3f5260e700], test_checkpoint_snapshot02.test_checkpoint_snapshot02.test_checkpoint_snapshot_with_txnid_and_timestamp(row_integer.crash_restart), statlog-server: [WT_VERB_DEFAULT][ERROR]: __statlog_server, 597: statistics log server error: No space left on device
[1661976154:254869][44383:0x7f3f5260e700], test_checkpoint_snapshot02.test_checkpoint_snapshot02.test_checkpoint_snapshot_with_txnid_and_timestamp(row_integer.crash_restart), statlog-server: [WT_VERB_DEFAULT][ERROR]: __statlog_server, 597: the process must exit and restart: WT_PANIC: WiredTiger library panic
[1661976154:254882][44383:0x7f3f5260e700], test_checkpoint_snapshot02.test_checkpoint_snapshot02.test_checkpoint_snapshot_with_txnid_and_timestamp(row_integer.crash_restart), statlog-server: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 28: aborting WiredTiger library
Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_spinlock_pthread_adaptive_test_fc5838e9095b279533f151b03dc31ff2f3756b00_22_08_31_18_09_21/0?type=T#L19402

Evergreen YAML Script Failure
Command failed: error opening target archive file /data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger.tgz: open /data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger.tgz: no space left on device
Command failed: missing file '/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger.tgz': opening file '/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger.tgz': open /data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger.tgz: no such file or directory
Command failed: error opening target archive file /data/mci/b54f183ca19ff42b15ceea33c627999a/wt-hang-analyzer.tgz: open /data/mci/b54f183ca19ff42b15ceea33c627999a/wt-hang-analyzer.tgz: no space left on device
Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_spinlock_pthread_adaptive_test_fc5838e9095b279533f151b03dc31ff2f3756b00_22_08_31_18_09_21/0?type=T#L19408

logs

duplicates

WT-11359 Update spinlock tasks to limit disk usage

Closed

Assignee:: [DO NOT USE] Backlog - Storage Engines Team
Reporter:: xgen-buildbaron-user
Votes:: 0 Vote for this issue
Watchers:: 5 Start watching this issue

Created:: Sep 08 2022 03:01:34 PM UTC
Updated:: Sep 21 2023 02:09:29 AM UTC
Resolved:: Sep 21 2023 02:09:12 AM UTC

Details

Description

Attachments

Issue Links

Activity

People

Dates