Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-9303

Fix checkpoint failure in verify operation - no space on device

    XMLWordPrintableJSON

Details

    • Build Failure
    • Status: Closed
    • Major - P3
    • Resolution: Works as Designed
    • None
    • None
    • None
    • 5
    • Storage Engines - 2022-06-27

    Description

      spinlock-pthread-adaptive-test on ubuntu2004

      Host: i-0d1b10211b2505929
      Project: wiredtiger
      Commit: a1d0eda7

      Task Logs:

      spinlock-pthread-adaptive-test task_log

      Logs:

      =======================================================
      ERROR: test_alter02.test_alter02.test_alter02_log(s3.conn-always-logged.lsm.create-logged.reopen) (subunit.RemotedTestCase)
      test_alter02.test_alter02.test_alter02_log(s3.conn-always-logged.lsm.create-logged.reopen)
      ----------------------------------------------------------------------
      testtools.testresult.real._StringException: Traceback (most recent call last):
        File "/data/mci/45b522c9b6701535dde8c9578c525162/wiredtiger/test/suite/wttest.py", line 519, in setUp
          os.makedirs(self.testdir)
        File "/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/python3-v3.9MG/lib/python3.9/os.py", line 225, in makedirs
          mkdir(name, mode)
      OSError: [Errno 28] No space left on device: 'WT_TEST/test_alter02.test_alter02.test_alter02_log_s3.conn-always-logged.lsm.create-logged.reopen'
      ======================================================================
      ERROR: test_a
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_spinlock_pthread_adaptive_test_a1d0eda743a51dd607743af19eef48773e650640_22_05_09_15_39_55/0?type=T#L20160
       
      Verbose Error message
      [1652117637:605608][53148:0x7fc126818740], test_alter04.test_alter04.test_alter04_cache(non_tiered.table-cg.200K.reopen.cache), file:WiredTiger.wt, WT_CONNECTION.close: [WT_VERB_DEFAULT][ERROR]: __wt_block_checkpoint_resolve, 928: the process must exit and restart: WT_PANIC: WiredTiger library panic
      [1652117637:605669][53148:0x7fc126818740], test_alter04.test_alter04.test_alter04_cache(non_tiered.table-cg.200K.reopen.cache), file:WiredTiger.wt, WT_CONNECTION.close: [WT_VERB_DEFAULT][ERROR]: __wt_conn_dhandle_discard_single, 871: Final close of file:WiredTiger.wt failed: WT_PANIC: WiredTiger library panic
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_spinlock_pthread_adaptive_test_a1d0eda743a51dd607743af19eef48773e650640_22_05_09_15_39_55/0?type=T#L102920
       
      Evergreen YAML Script Failure
      Command failed: shell script encountered problem: exit code 1
      Command failed: shell script encountered problem: exit code 1
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_spinlock_pthread_adaptive_test_a1d0eda743a51dd607743af19eef48773e650640_22_05_09_15_39_55/0?type=T#L102964
       
      Verbose Error message
      [1652117637:604700][53154:0x7fc1222ca700], log-server: [WT_VERB_DEFAULT][ERROR]: __posix_open_file, 801: ././WiredTigerTmplog.0000000002: handle-open: open: No space left on device
      [1652117637:607217][53154:0x7fc1222ca700], log-server: [WT_VERB_DEFAULT][ERROR]: __log_prealloc_once, 510: log pre-alloc server error: No space left on device
      [1652117637:607254][53154:0x7fc1222ca700], log-server: [WT_VERB_DEFAULT][ERROR]: __log_server, 914: log server error: No space left on device
      [1652117637:607275][53154:0x7fc1222ca700], log-server: [WT_VERB_DEFAULT][ERROR]: __log_server, 914: the process must exit and restart: WT_PANIC: WiredTiger library panic
      [1652117637:608163][53154:0x7fc126818740], file:WiredTiger.wt, close_ckpt: [WT_VERB_DEFAULT][ERROR]: __posix_open_file, 801: ./WiredTiger.turtle.set: handle-open: open: No space left on device
      [1652117637:608205][53154:0x7fc126818740], file:WiredTiger.wt, close_ckpt: [WT_VERB_DEFAULT][ERROR]: __wt_block_checkpoint_resolve, 928: WiredTiger.wt: the checkpoint failed, the system must restart: Invalid argument
      [1652117637:608229][53154:0x7fc126818740], close_ckpt: [WT_VERB_DEFAULT][ERROR]: __wt_cursor_set_key, 289: failed to set key: WT_PANIC: WiredTiger library panic
      [1652117637:608243][53154:0x7fc126818740], close_ckpt: [WT_VERB_DEFAULT][ERROR]: __meta_track_unroll, 213: metadata unroll update system:checkpoint_base_write_gen to base_write_gen=16: WT_PANIC: WiredTiger library panic
      [1652117637:608257][53154:0x7fc126818740], close_ckpt: [WT_VERB_DEFAULT][ERROR]: __wt_cursor_set_key, 289: failed to set key: WT_PANIC: WiredTiger library panic
      [1652117637:608268][53154:0x7fc126818740], close_ckpt: [WT_VERB_DEFAULT][ERROR]: __meta_track_unroll, 213: metadata unroll update system:checkpoint_snapshot to snapshot_min=1,snapshot_max=1,snapshot_count=0,checkpoint_time=1652117637,write_gen=16: WT_PANIC: WiredTiger library panic
      [1652117637:608282][53154:0x7fc126818740], close_ckpt: [WT_VERB_DEFAULT][ERROR]: __wt_meta_track_off, 334: failed to apply or unroll all tracked operations: WT_PANIC: WiredTiger library panic
      [1652117637:608378][53154:0x7fc126818740], WT_CONNECTION.close: [WT_VERB_DEFAULT][ERROR]: __conn_close, 1177: failure during close, disabling further writes: WT_PANIC: WiredTiger library panic
      [1652117637:608458][53154:0x7fc11eac3700], eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_evict_thread_run, 324: cache eviction thread error: WT_PANIC: WiredTiger library panic
      [1652117637:608506][53154:0x7fc121ac9700], eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_evict_thread_run, 324: cache eviction thread error: WT_PANIC: WiredTiger library panic
      [1652117637:608556][53154:0x7fc1202c6700], eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_evict_thread_run, 324: cache eviction thread error: WT_PANIC: WiredTiger library panic
      [1652117637:608627][53154:0x7fc1212c8700], eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_evict_thread_run, 324: cache eviction thread error: WT_PANIC: WiredTiger library panic
      [1652117637:608661][53154:0x7fc11f2c4700], eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_evict_thread_run, 324: cache eviction thread error: WT_PANIC: WiredTiger library panic
      [1652117637:608526][53154:0x7fc11fac5700], eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_evict_thread_run, 324: cache eviction thread error: WT_PANIC: WiredTiger library panic
      [1652117637:608595][53154:0x7fc120ac7700], eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_evict_thread_run, 324: cache eviction thread error: WT_PANIC: WiredTiger library panic
      [1652117637:609358][53154:0x7fc1242ce700], log-wrlsn-server: [WT_VERB_DEFAULT][ERROR]: __wt_log_slot_switch, 416: log slot switch fatal error: WT_PANIC: WiredTiger library panic
      [1652117637:609588][53154:0x7fc1242ce700], log-wrlsn-server: [WT_VERB_DEFAULT][ERROR]: __log_wrlsn_server, 824: log wrlsn server error: WT_PANIC: WiredTiger library panic
      [1652117637:609716][53154:0x7fc126818740], WT_CONNECTION.close: [WT_VERB_DEFAULT][ERROR]: __wt_cache_destroy, 364: cache server: exiting with 4 pages in memory and 0 pages evicted
      [1652117637:604507][53148:0x7fc126818740], test_alter04.test_alter04.test_alter04_cache(non_tiered.table-cg.200K.reopen.cache), file:WiredTiger.wt, WT_CONNECTION.close: [WT_VERB_DEFAULT][ERROR]: __posix_open_file, 801: ./WiredTiger.turtle.set: handle-open: open: No space left on device
      [1652117637:605576][53148:0x7fc126818740], test_alter04.test_alter04.test_alter04_cache(non_tiered.table-cg.200K.reopen.cache), file:WiredTiger.wt, WT_CONNECTION.close: [WT_VERB_DEFAULT][ERROR]: __wt_block_checkpoint_resolve, 928: WiredTiger.wt: the checkpoint failed, the system must restart: Invalid argument
      [1652117637:605608][53148:0x7fc126818740], test_alter04.test_alter04.test_alter04_cache(non_tiered.table-cg.200K.reopen.cache), file:WiredTiger.wt, WT_CONNECTION.close: [WT_VERB_DEFAULT][ERROR]: __wt_block_checkpoint_resolve, 928: the process must exit and restart: WT_PANIC: WiredTiger library panic
      [1652117637:605669][53148:0x7fc126818740], test_alter04.test_alter04.test_alter04_cache(non_tiered.table-cg.200K.reopen.cache), file:WiredTiger.wt, WT_CONNECTION.close: [WT_VERB_DEFAULT][ERROR]: __wt_conn_dhandle_discard_single, 871: Final close of file:WiredTiger.wt failed: WT_PANIC: WiredTiger library panic
      [1652117637:602880][53150:0x7fc126818740], file:WiredTiger.wt, WT_CONNECTION.close: [WT_VERB_DEFAULT][ERROR]: __posix_open_file, 801: ./WiredTiger.turtle.set: handle-open: open: No space left on device
      [1652117637:602969][53150:0x7fc126818740], file:WiredTiger.wt, WT_CONNECTION.close: [WT_VERB_DEFAULT][ERROR]: __wt_block_checkpoint_resolve, 928: WiredTiger.wt: the checkpoint failed, the system must restart: Invalid argument
      [1652117637:602988][53150:0x7fc126818740], file:WiredTiger.wt, WT_CONNECTION.close: [WT_VERB_DEFAULT][ERROR]: __wt_block_checkpoint_resolve, 928: the process must exit and restart: WT_PANIC: WiredTiger library panic
      [1652117637:603054][53150:0x7fc126818740], file:WiredTiger.wt, WT_CONNECTION.close: [WT_VERB_DEFAULT][ERROR]: __wt_conn_dhandle_discard_single, 871: Final close of file:WiredTiger.wt failed: WT_PANIC: WiredTiger library panic
      [1652117637:603307][53150:0x7fc1222ca700], log-wrlsn-server: [WT_VERB_DEFAULT][ERROR]: __log_wrlsn_server, 824: log wrlsn server error: WT_PANIC: WiredTiger library panic
      [1652117637:601947][53156:0x7fc126818740], connection: [WT_VERB_DEFAULT][ERROR]: __posix_open_file, 801: ./WiredTiger.wt: handle-open: open: No space left on device
      [1652117637:603384][53156:0x7fc126818740], file:WiredTiger.wt, connection: [WT_VERB_DEFAULT][ERROR]: __posix_open_file, 801: ./WiredTiger.wt: handle-open: open: No such file or directory
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_spinlock_pthread_adaptive_test_a1d0eda743a51dd607743af19eef48773e650640_22_05_09_15_39_55/0?type=T#L103020
       
      Evergreen YAML Script Failure
      Command failed: error opening target archive file /data/mci/45b522c9b6701535dde8c9578c525162/wiredtiger.tgz: open /data/mci/45b522c9b6701535dde8c9578c525162/wiredtiger.tgz: no space left on device
      Command failed: missing file '/data/mci/45b522c9b6701535dde8c9578c525162/wiredtiger.tgz': problem opening file /data/mci/45b522c9b6701535dde8c9578c525162/wiredtiger.tgz: open /data/mci/45b522c9b6701535dde8c9578c525162/wiredtiger.tgz: no such file or directory
      Command failed: error opening target archive file /data/mci/45b522c9b6701535dde8c9578c525162/wt-hang-analyzer.tgz: open /data/mci/45b522c9b6701535dde8c9578c525162/wt-hang-analyzer.tgz: no space left on device
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_spinlock_pthread_adaptive_test_a1d0eda743a51dd607743af19eef48773e650640_22_05_09_15_39_55/0?type=T#L103078
      

      logs

      Attachments

        Activity

          People

            keith.smith@mongodb.com Keith Smith
            Xgen-BuildBaron-User xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: