Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-9843

Investigate out of space failures in spinlock-pthread-adaptive-test on ubuntu2004 [wiredtiger @ fc5838e9]

    XMLWordPrintable

Details

    • Build Failure
    • Status: Open
    • Minor - P4
    • Resolution: Unresolved
    • None
    • None
    • None
    • 1
    • StorEng - Defined Pipeline

    Description

      This test failed part-way through the python test suite with an out of space error. All of the remaining python tests at that point failed for the same reason. The goal on this ticket should be to understand why this particular test run ran out of disk space and, if needed, adjust our evergreen testing to make this less likely.

      spinlock-pthread-adaptive-test on ubuntu2004

      Host: i-0ea55d06dfca15d54
      Project: wiredtiger
      Commit: fc5838e9

      Task Logs:

      spinlock-pthread-adaptive-test task_log

      Logs:

      t connection during test 'test_checkpoint_snapshot02.test_checkpoint_snapshot02.test_checkpoint_snapshot_with_txnid_and_timestamp(column_fix.backup)'
      ======================================================================
      ERROR: test_compat01.test_compat01.test_restart(old2.old_patch) (subunit.RemotedTestCase)
      test_compat01.test_compat01.test_restart(old2.old_patch)
      ----------------------------------------------------------------------
      testtools.testresult.real._StringException: Traceback (most recent call last):
        File "/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger/test/suite/wttest.py", line 522, in setUp
          os.makedirs(self.testdir)
        File "/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/python3-v3.9MG/lib/python3.9/os.py", line 225, in makedirs
          mkdir(name, mode)
      OSError: [Errno 28] No space left on device: 'WT_TEST/test_compat01.test_compat01.test_restart_old2.old_patch'
      ======================================================================
      ERROR: test_compat01.test_compat01.test_restart(old_patch2.30_patch) (subunit.RemotedTestCase)
      test_compat01.test_compat01.test_restart(old_patch2.30_patch)
      ----------------------------------------------------------------------
      testtools.testresult.real._StringException: Traceback (most recent call last):
        File "/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger/test/suite/wttest.py", line 522, in setUp
          os.makedirs(self.testdir)
        File "/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/python3-v3.9MG/lib/python3.9/os.py", line 225, in makedirs
          mkdir(name, mode)
      OSError: [Errno 28] No space left on device: 'WT_TEST/test_compat01.test_compat01.test_restart_old_patch2.30_patch'
      ======================================================================
      ERROR: test_compat02.test_compat02.test_compat02(def.100_rel.def_min.110_max.basecfg_false) (subunit.RemotedTestCase)
      test_compat02.test_compat02.test_compat02(def.100_rel.def_min.110_max.basecfg_false)
      ----------------------------------------------------------------------
      testtools.testresult.real._StringException: Traceback (most recent call last):
        File "/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger/test/suite/wttest.py", line 522, in setUp
          os.makedirs(self.testdir)
        File "/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/python3-v3.9MG/lib/python3.9/os.py", line 225, in makedirs
          mkdir(name, mode)
      OSError: [Errno 28] No space left on device: 'WT_TEST/test_compat02.test_compat02.test_compat02_def.100_rel.def_min.110_max.basecfg_false'
      ======================================================================
      ERROR: test_compat02.test_compat02.test_compat02(def.26_rel.100_min.def_max.basecfg_false) (subunit.RemotedTestCase)
      test_compat02.test_compat02.test_compat02(def.26_rel.100_min.def_max.basecfg_false)
      ----------------------------------------------------------------------
      testtools.testresult.real._StringException: Traceback (most recent call last):
        File "/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger/test/suite/wttest.py", line 522, in setUp
          os.makedirs(self.testdir)
        File "/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/python3-v3.9MG/lib/python3.9/os.py", line 225, in makedirs
          mkdir(name, mode)
      OSError: [Errno 28] No space left on device: 'WT_TEST/test_compat02.test_compat02.test_compat02_def.26_rel.100_min.def_max.basecfg_false'
      ======================================================================
      ERROR: test_compat02.test_compat02.test_compat02(110.31_rel.33_min.33_max.basecfg_false) (subunit.RemotedTestCase)
      test_compat02.test_compat02.test_compat02(110.31_rel.33_min.33_max.basecfg_false)
      ----------------------------------------------------------------------
      testtools.testresult.real._StringException: Traceback (most recent call last):
        File "/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger/test/suite/wttest.py", line 522, in setUp
          os.makedirs(self.testdir)
        File "/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/python3-v3.9MG/lib/python3.9/os.py", line 225, in makedirs
          mkdir(name, mode)
      OSError: [Errno 28] No space left on device: 'WT_TEST/test_compat02.test_compat02.test_compat02_110.31_rel.33_min.33_max.basecfg_false'
      ======================================================================
      ERROR: test_compat02.test_compat02.test_compat02(110.26_rel.30_min.future_max.basecfg_true) (subunit.RemotedTestCase)
      test_compat02.test_compat02.test_compat02(110.26_rel.30_min.future_max.basecfg_true)
      ----------------------------------------------------------------------
      testtools.testresult.real._StringException: Traceback (most recent call last):
        File "/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger/test/suite/wttest.py", line 522, in setUp
          os.makedirs(self.testdir)
        File "/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/python3-v3.9MG/lib/python3.9/os.py", line 225, in makedirs
          mkdir(name, mode)
      OSError: [Errno 28] No space left on device: 'WT_TEST/test_compat02.test_compat02.test_compat02_110.26_rel.30_min.future_max.basecfg_true'
      ======================================================================
      ERROR: test_compat02.test_compat02.test_compat02(100.30_rel.future_min.32_max.basecfg_false) (subunit.RemotedTestCase)
      test_compat02.test_compat02.test_compat02(100.30_rel.future_min.32_max.basecfg_false)
      ----------------------------------------------------------------------
      testtools.testresult.real._StringException: Traceback (most recent call last):
        File "/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger/test/suite/wttest.py", line 522, in setUp
          os.makedirs(self.testdir)
        File "/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/python3-v3.9MG/lib/python3.9/os.py", line 225, in makedirs
          mkdir(name, mode)
      OSError: [Errno 28] No space left on device: 'WT_TEST/test_compat02.test_compat02.test_compat02_100.30_rel.future_min.32_max.basecfg_false'
      ======================================================================
      ERROR: test_compat02.test_compat02.test_compat02(33.100_rel.26_min.30_max.basecfg_false) (subunit.RemotedTestCase)
      test_compat02.test_compat02.test_compat02(33.100_rel.26_min.30_max.basecfg_false)
      ----------------------------------------------------------------------
      testtools.testresult.real._StringException: Traceback (most recent call last):
        File "/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger/test/suite/wttest.py", line 522, in setUp
          os.makedirs(self.testdir)
        File "/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/python3-v3.9MG/lib/python3.9/os.py", line 225, in makedirs
          mkdir(name, mode)
      OSError: [Errno 28] No space left on device: 'WT_TEST/test_compat02.test_compat02.test_compat02_33.100_rel.26_min.30_max.basecfg_false'
      ======================================================================
      ERROR: test_compat02.test_compat02.test_compat02(32.110_rel.future_min.def_max.basecfg_true) (subunit.RemotedTestCase)
      test_compat02.test_compat02.test_compat02(32.110_rel.future_min.def_max.basecfg_true)
      ----------------------------------------------------------------------
      testtools.testresul
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_spinlock_pthread_adaptive_test_fc5838e9095b279533f151b03dc31ff2f3756b00_22_08_31_18_09_21/0?type=T#L8213
       
      Evergreen YAML Script Failure
      Command failed: shell script encountered problem: exit code 1
       
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_spinlock_pthread_adaptive_test_fc5838e9095b279533f151b03dc31ff2f3756b00_22_08_31_18_09_21/0?type=T#L19382
       
      Verbose Error message
      [1661976154:254767][44383:0x7f3f5260e700], test_checkpoint_snapshot02.test_checkpoint_snapshot02.test_checkpoint_snapshot_with_txnid_and_timestamp(row_integer.crash_restart), statlog-server: [WT_VERB_DEFAULT][ERROR]: __posix_file_write, 611: BACKUP/./WiredTigerStat.31.20: handle-write: pwrite: failed to write 7136 bytes at offset 663552: No space left on device
      [1661976154:254852][44383:0x7f3f5260e700], test_checkpoint_snapshot02.test_checkpoint_snapshot02.test_checkpoint_snapshot_with_txnid_and_timestamp(row_integer.crash_restart), statlog-server: [WT_VERB_DEFAULT][ERROR]: __statlog_server, 597: statistics log server error: No space left on device
      [1661976154:254869][44383:0x7f3f5260e700], test_checkpoint_snapshot02.test_checkpoint_snapshot02.test_checkpoint_snapshot_with_txnid_and_timestamp(row_integer.crash_restart), statlog-server: [WT_VERB_DEFAULT][ERROR]: __statlog_server, 597: the process must exit and restart: WT_PANIC: WiredTiger library panic
      [1661976154:254882][44383:0x7f3f5260e700], test_checkpoint_snapshot02.test_checkpoint_snapshot02.test_checkpoint_snapshot_with_txnid_and_timestamp(row_integer.crash_restart), statlog-server: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 28: aborting WiredTiger library
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_spinlock_pthread_adaptive_test_fc5838e9095b279533f151b03dc31ff2f3756b00_22_08_31_18_09_21/0?type=T#L19402
       
      Evergreen YAML Script Failure
      Command failed: error opening target archive file /data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger.tgz: open /data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger.tgz: no space left on device
      Command failed: missing file '/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger.tgz': opening file '/data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger.tgz': open /data/mci/b54f183ca19ff42b15ceea33c627999a/wiredtiger.tgz: no such file or directory
      Command failed: error opening target archive file /data/mci/b54f183ca19ff42b15ceea33c627999a/wt-hang-analyzer.tgz: open /data/mci/b54f183ca19ff42b15ceea33c627999a/wt-hang-analyzer.tgz: no space left on device
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_spinlock_pthread_adaptive_test_fc5838e9095b279533f151b03dc31ff2f3756b00_22_08_31_18_09_21/0?type=T#L19408
      

      logs

      Attachments

        Activity

          People

            backlog-server-storage-engines Backlog - Storage Engines Team
            Xgen-BuildBaron-User xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated: