Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-9843

Investigate out of space failures in spinlock-pthread-adaptive-test on ubuntu2004 [wiredtiger @ fc5838e9]

XMLWordPrintableJSON

    • Type: Icon: Build Failure Build Failure
    • Resolution: Duplicate
    • Priority: Icon: Minor - P4 Minor - P4
    • None
    • Affects Version/s: None
    • Component/s: None
    • NachoCheese - 2023-10-03

      This test failed part-way through the python test suite with an out of space error. All of the remaining python tests at that point failed for the same reason. The goal on this ticket should be to understand why this particular test run ran out of disk space and, if needed, adjust our evergreen testing to make this less likely.

      spinlock-pthread-adaptive-test on ubuntu2004

      Host: i-0ea55d06dfca15d54
      Project: wiredtiger
      Commit: fc5838e9

      Task Logs:

      spinlock-pthread-adaptive-test task_log

      Logs:

      t connection during test 'test_checkpoint_snapshot02.test_checkpoint_snapshot02.test_checkpoint_snapshot_with_txnid_and_timestamp(column_fix.backup)'
======================================================================
ERROR: test_compat01.test_compat01.test_restart(old2.old_patch) (subunit.RemotedTestCase)
test_compat01.test_compat01.test_restart(old2.old_patch)
----------------------------------------------------------------------
testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger/test/suite/wttest.py", line 522, in setUp
    os.makedirs(self.testdir)
  File "/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/python3-v3.9MG/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
OSError: [Errno 28] No space left on device: 'WT_TEST/test_compat01.test_compat01.test_restart_old2.old_patch'
======================================================================
ERROR: test_compat01.test_compat01.test_restart(old_patch2.30_patch) (subunit.RemotedTestCase)
test_compat01.test_compat01.test_restart(old_patch2.30_patch)
----------------------------------------------------------------------
testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger/test/suite/wttest.py", line 522, in setUp
    os.makedirs(self.testdir)
  File "/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/python3-v3.9MG/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
OSError: [Errno 28] No space left on device: 'WT_TEST/test_compat01.test_compat01.test_restart_old_patch2.30_patch'
======================================================================
ERROR: test_compat02.test_compat02.test_compat02(def.100_rel.def_min.110_max.basecfg_false) (subunit.RemotedTestCase)
test_compat02.test_compat02.test_compat02(def.100_rel.def_min.110_max.basecfg_false)
----------------------------------------------------------------------
testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger/test/suite/wttest.py", line 522, in setUp
    os.makedirs(self.testdir)
  File "/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/python3-v3.9MG/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
OSError: [Errno 28] No space left on device: 'WT_TEST/test_compat02.test_compat02.test_compat02_def.100_rel.def_min.110_max.basecfg_false'
======================================================================
ERROR: test_compat02.test_compat02.test_compat02(def.26_rel.100_min.def_max.basecfg_false) (subunit.RemotedTestCase)
test_compat02.test_compat02.test_compat02(def.26_rel.100_min.def_max.basecfg_false)
----------------------------------------------------------------------
testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger/test/suite/wttest.py", line 522, in setUp
    os.makedirs(self.testdir)
  File "/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/python3-v3.9MG/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
OSError: [Errno 28] No space left on device: 'WT_TEST/test_compat02.test_compat02.test_compat02_def.26_rel.100_min.def_max.basecfg_false'
======================================================================
ERROR: test_compat02.test_compat02.test_compat02(110.31_rel.33_min.33_max.basecfg_false) (subunit.RemotedTestCase)
test_compat02.test_compat02.test_compat02(110.31_rel.33_min.33_max.basecfg_false)
----------------------------------------------------------------------
testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger/test/suite/wttest.py", line 522, in setUp
    os.makedirs(self.testdir)
  File "/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/python3-v3.9MG/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
OSError: [Errno 28] No space left on device: 'WT_TEST/test_compat02.test_compat02.test_compat02_110.31_rel.33_min.33_max.basecfg_false'
======================================================================
ERROR: test_compat02.test_compat02.test_compat02(110.26_rel.30_min.future_max.basecfg_true) (subunit.RemotedTestCase)
test_compat02.test_compat02.test_compat02(110.26_rel.30_min.future_max.basecfg_true)
----------------------------------------------------------------------
testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger/test/suite/wttest.py", line 522, in setUp
    os.makedirs(self.testdir)
  File "/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/python3-v3.9MG/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
OSError: [Errno 28] No space left on device: 'WT_TEST/test_compat02.test_compat02.test_compat02_110.26_rel.30_min.future_max.basecfg_true'
======================================================================
ERROR: test_compat02.test_compat02.test_compat02(100.30_rel.future_min.32_max.basecfg_false) (subunit.RemotedTestCase)
test_compat02.test_compat02.test_compat02(100.30_rel.future_min.32_max.basecfg_false)
----------------------------------------------------------------------
testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger/test/suite/wttest.py", line 522, in setUp
    os.makedirs(self.testdir)
  File "/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/python3-v3.9MG/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
OSError: [Errno 28] No space left on device: 'WT_TEST/test_compat02.test_compat02.test_compat02_100.30_rel.future_min.32_max.basecfg_false'
======================================================================
ERROR: test_compat02.test_compat02.test_compat02(33.100_rel.26_min.30_max.basecfg_false) (subunit.RemotedTestCase)
test_compat02.test_compat02.test_compat02(33.100_rel.26_min.30_max.basecfg_false)
----------------------------------------------------------------------
testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger/test/suite/wttest.py", line 522, in setUp
    os.makedirs(self.testdir)
  File "/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/python3-v3.9MG/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
OSError: [Errno 28] No space left on device: 'WT_TEST/test_compat02.test_compat02.test_compat02_33.100_rel.26_min.30_max.basecfg_false'
======================================================================
ERROR: test_compat02.test_compat02.test_compat02(32.110_rel.future_min.def_max.basecfg_true) (subunit.RemotedTestCase)
test_compat02.test_compat02.test_compat02(32.110_rel.future_min.def_max.basecfg_true)
----------------------------------------------------------------------
testtools.testresul
Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_spinlock_pthread_adaptive_test_fc5838e9095b279533f151b03dc31ff2f3756b00_22_08_31_18_09_21/0?type=T#L8213

Evergreen YAML Script Failure
Command failed: shell script encountered problem: exit code 1

Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_spinlock_pthread_adaptive_test_fc5838e9095b279533f151b03dc31ff2f3756b00_22_08_31_18_09_21/0?type=T#L19382

Verbose Error message
[1661976154:254767][44383:0x7f3f5260e700], test_checkpoint_snapshot02.test_checkpoint_snapshot02.test_checkpoint_snapshot_with_txnid_and_timestamp(row_integer.crash_restart), statlog-server: [WT_VERB_DEFAULT][ERROR]: __posix_file_write, 611: BACKUP/./WiredTigerStat.31.20: handle-write: pwrite: failed to write 7136 bytes at offset 663552: No space left on device
[1661976154:254852][44383:0x7f3f5260e700], test_checkpoint_snapshot02.test_checkpoint_snapshot02.test_checkpoint_snapshot_with_txnid_and_timestamp(row_integer.crash_restart), statlog-server: [WT_VERB_DEFAULT][ERROR]: __statlog_server, 597: statistics log server error: No space left on device
[1661976154:254869][44383:0x7f3f5260e700], test_checkpoint_snapshot02.test_checkpoint_snapshot02.test_checkpoint_snapshot_with_txnid_and_timestamp(row_integer.crash_restart), statlog-server: [WT_VERB_DEFAULT][ERROR]: __statlog_server, 597: the process must exit and restart: WT_PANIC: WiredTiger library panic
[1661976154:254882][44383:0x7f3f5260e700], test_checkpoint_snapshot02.test_checkpoint_snapshot02.test_checkpoint_snapshot_with_txnid_and_timestamp(row_integer.crash_restart), statlog-server: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 28: aborting WiredTiger library
Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_spinlock_pthread_adaptive_test_fc5838e9095b279533f151b03dc31ff2f3756b00_22_08_31_18_09_21/0?type=T#L19402

Evergreen YAML Script Failure
Command failed: error opening target archive file /data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger.tgz: open /data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger.tgz: no space left on device
Command failed: missing file '/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger.tgz': opening file '/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger.tgz': open /data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger.tgz: no such file or directory
Command failed: error opening target archive file /data/mci/b54f183ca19ff42b15ceea33c627999a/wt-hang-analyzer.tgz: open /data/mci/b54f183ca19ff42b15ceea33c627999a/wt-hang-analyzer.tgz: no space left on device
Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_spinlock_pthread_adaptive_test_fc5838e9095b279533f151b03dc31ff2f3756b00_22_08_31_18_09_21/0?type=T#L19408

      logs

            Assignee:
            backlog-server-storage-engines [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Xgen-BuildBaron-User xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: