Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-9567

Cache stuck issue recovery-stress-test-3 on ubuntu2004-stress-tests [wiredtiger-mongo-v5.0 @ 09f3e7c4]

    • Type: Icon: Build Failure Build Failure
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • 3
    • Storage Engines - 2022-11-28

      recovery-stress-test-3 on ubuntu2004-stress-tests

      Host: i-0b79e1b881312669a
      Project: wiredtiger-mongo-v5.0
      Commit: 09f3e7c4

      Task Logs:

      recovery-stress-test-3 task_log

      Logs:

      Cache Stuck Failures
      transaction state dump
      current ID: 11748
      last running ID: 11482
      metadata_pinned ID: 11211
      oldest ID: 11211
      durable timestamp: (0, 11574)
      oldest timestamp: (0, 10764)
      pinned timestamp: (0, 10764)
      stable timestamp: (0, 10764)
      has_durable_timestamp: yes
      has_oldest_timestamp: yes
      has_pinned_timestamp: yes
      has_stable_timestamp: yes
      oldest_is_pinned: yes
      stable_is_pinned: yes
      checkpoint running: yes
      checkpoint generation: 2
      checkpoint pinned ID: 0
      checkpoint txn ID: 0
      session count: 104
      Transaction state of active sessions:
      ID: 11493, pinned ID: 11211, metadata pinned ID: 11245, name: EMPTY
      transaction id: 11493, mod count: 1, snap min: 11211, snap max: 11492, snapshot count: 76, commit_timestamp: (0, 11324), durable_timestamp: (0, 11324), first_commit_timestamp: (0, 11323), prepare_timestamp: (0, 0), pinned_durable_timestamp: (0, 11323), read_timestamp: (0, 0), checkpoint LSN: [0][0], full checkpoint: false, rollback reason: , flags: 0x0000181c, isolation: WT_ISO_SNAPSHOT
      ID: 11741, pinned ID: 11482, metadata pinned ID: 0, name: EMPTY
      transaction id: 11741, mod count: 1, snap min: 11482, snap max: 11740, snapshot count: 3, commit_timestamp: (0, 11568), durable_timestamp: (0, 11568), first_commit_timestamp: (0, 11567), prepare_timestamp: (0, 0), pinned_durable_timestamp: (0, 11567), read_timestamp: (0, 0), checkpoint LSN: [0][0], full checkpoint: false, rollback reason: , flags: 0x0000181c, isolation: WT_ISO_SNAPSHOT
      ID: 11482, pinned ID: 11211, metadata pinned ID: 11245, name: EMPTY
      transaction id: 11482, mod count: 1, snap min: 11211, snap max: 11482, snapshot count: 75, commit_timestamp: (0, 11320), durable_timestamp: (0, 11320), first_commit_timestamp: (0, 11319), prepare_timestamp: (0, 0), pinned_durable_timestamp: (0, 11319), read_timestamp: (0, 0), checkpoint LSN: [0][0], full checkpoint: false, rollback reason: , flags: 0x0000181c, isolation: WT_ISO_SNAPSHOT
      =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
      cache dump
      cache full: no
      cache clean check: no (21.888%)
      cache dirty check: yes (20.088%)
      cache updates check: no (5.859%)
      file:oplog.wt(<live>):
      internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      leaf: 76 pages, 2MB, 4/72 clean/dirty pages, 0/2/0 clean/dirty/updates MB, 0MB max page, 0MB max dirty page
      file:local.wt(<live>):
      internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      leaf: 72 pages, 2MB, 9/63 clean/dirty pages, 0/1/0 clean/dirty/updates MB, 0MB max page, 0MB max dirty page
      file:shadow.wt(<live>):
      internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      leaf: 73 pages, 2MB, 5/68 clean/dirty pages, 0/1/0 clean/dirty/updates MB, 0MB max page, 0MB max dirty page
      file:collection.wt(<live>):
      internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      leaf: 58 pages, 2MB, 6/52 clean/dirty pages, 0/2/1 clean/dirty/updates MB, 0MB max page, 0MB max dirty page
      file:WiredTigerHS.wt(<live>) eviction disabled at open:
      internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      leaf: 0 pages
      file:WiredTiger.wt(<live>):
      internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      leaf: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0/0 clean/dirty/updates MB, 0MB max page, 0MB max dirty page
      cache dump: total found: 9MB vs tracked inuse 8MB
      total dirty bytes: 8MB vs tracked dirty 8MB
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_mongo_v5.0_ubuntu2004_stress_tests_recovery_stress_test_3_09f3e7c4cf70047cdb9f53585b4d848b64a9bed7_22_07_08_02_21_18/0?type=T#L5833
      
      Compiler and other error messages
      [1657251670:377348][302108:0x7fc4737fe700], eviction-server: __wt_evict_thread_run, 324: cache eviction thread error: Connection timed out
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_mongo_v5.0_ubuntu2004_stress_tests_recovery_stress_test_3_09f3e7c4cf70047cdb9f53585b4d848b64a9bed7_22_07_08_02_21_18/0?type=T#L5887
      
      Failure keywords
      [1657251670:377358][302108:0x7fc4737fe700], eviction-server: __wt_evict_thread_run, 324: the process must exit and restart: WT_PANIC: WiredTiger library panic
      [1657251670:377364][302108:0x7fc4737fe700], eviction-server: __wt_abort, 28: aborting WiredTiger library
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_mongo_v5.0_ubuntu2004_stress_tests_recovery_stress_test_3_09f3e7c4cf70047cdb9f53585b4d848b64a9bed7_22_07_08_02_21_18/0?type=T#L5888
      
      C/CPP Suite Failure message
      test_timestamp_abort: FAILED: Child process 302108 abnormally exited: Invalid argument
      
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_mongo_v5.0_ubuntu2004_stress_tests_recovery_stress_test_3_09f3e7c4cf70047cdb9f53585b4d848b64a9bed7_22_07_08_02_21_18/0?type=T#L5893
      
      Failure keywords
      test_timestamp_abort: process aborting
      WiredTiger Error: aborting WiredTiger library
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_mongo_v5.0_ubuntu2004_stress_tests_recovery_stress_test_3_09f3e7c4cf70047cdb9f53585b4d848b64a9bed7_22_07_08_02_21_18/0?type=T#L5894
      
      Evergreen YAML Script Failure
      Command failed: shell script encountered problem: exit code 134
      Command failed: missing file '/data/mci/2450c95982814098213c5fd748fa999a/wt-hang-analyzer.tgz': problem opening file /data/mci/2450c95982814098213c5fd748fa999a/wt-hang-analyzer.tgz: open /data/mci/2450c95982814098213c5fd748fa999a/wt-hang-analyzer.tgz: no such file or directory
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_mongo_v5.0_ubuntu2004_stress_tests_recovery_stress_test_3_09f3e7c4cf70047cdb9f53585b4d848b64a9bed7_22_07_08_02_21_18/0?type=T#L5897
      

      logs

        1. log_structure.txt
          91 kB
        2. log_structure2.txt
          88 kB
        3. Screen Shot 2022-08-08 at 2.44.44 pm.png
          Screen Shot 2022-08-08 at 2.44.44 pm.png
          613 kB
        4. WiredTigerStat.08.03
          7.17 MB

            Assignee:
            backlog-server-storage-engines [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Xgen-BuildBaron-User xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: