Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-10169

format cache stuck after compact

    • Type: Icon: Build Failure Build Failure
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None

      format-stress-sanitizer-test-ppc-ubuntu on ubuntu1804-ppc

      Host: ubuntu1804-ppc-6.pic.build.10gen.cc
      Project: wiredtiger-mongo-v6.1
      Commit: 9f60092c

      Task Logs:

      format-stress-sanitizer-test-ppc-ubuntu task_log

      Logs:

      g: yes
          0xfdc45908d60:checkpoint generation: 13
          0xfdc45908d60:checkpoint pinned ID: 5037
          0xfdc45908d60:checkpoint txn ID: 5037
          0xfdc45908d60:session count: 25
          0xfdc45908d60:Transaction state of active sessions:
          0xfdc45908d60:ID: 0, pinned ID: 5290, metadata pinned ID: 5037, name: WT_CURSOR.insert
          0xfdc45908d60:transaction id: 0, mod count: 0, snap min: 5037, snap max: 5290, snapshot count: 1, commit_timestamp: (0, 0), durable_timestamp: (0, 10535), first_commit_timestamp: (0, 0), prepare_timestamp: (0, 0), pinned_durable_timestamp: (0, 0), read_timestamp: (0, 10518), checkpoint LSN: [0][0], full checkpoint: false, rollback reason: , flags: 0x00085008, isolation: WT_ISO_SNAPSHOT
          0xfdc45908d60:ID: 0, pinned ID: 5290, metadata pinned ID: 5037, name: WT_CURSOR.search_near
          0xfdc45908d60:transaction id: 0, mod count: 0, snap min: 5037, snap max: 5290, snapshot count: 1, commit_timestamp: (0, 0), durable_timestamp: (0, 10536), first_commit_timestamp: (0, 0), prepare_timestamp: (0, 0), pinned_durable_timestamp: (0, 0), read_timestamp: (0, 10527), checkpoint LSN: [0][0], full checkpoint: false, rollback reason: , flags: 0x00005008, isolation: WT_ISO_SNAPSHOT
          0xfdc45908d60:ID: 0, pinned ID: 5290, metadata pinned ID: 5037, name: WT_CURSOR.search
          0xfdc45908d60:transaction id: 0, mod count: 0, snap min: 5037, snap max: 5290, snapshot count: 1, commit_timestamp: (0, 0), durable_timestamp: (0, 10528), first_commit_timestamp: (0, 0), prepare_timestamp: (0, 0), pinned_durable_timestamp: (0, 0), read_timestamp: (0, 10503), checkpoint LSN: [0][0], full checkpoint: false, rollback reason: , flags: 0x00005008, isolation: WT_ISO_SNAPSHOT
          0xfdc45908d60:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
          0xfdc45908d60:cache dump
          0xfdc45908d60:cache full: no
          0xfdc45908d60:cache clean check: no (69.942%)
          0xfdc45908d60:cache dirty check: yes (20.529%)
          0xfdc45908d60:cache updates check: no (1.693%)
          0xfdc45908d60:file:T00002.wt(<live>):
          0xfdc45908d60:internal: 6 pages, 568.30 KB, 3/3 clean/dirty pages, 352.80/215.50 clean / dirty KB, 118.85 KB max page, 116.66 KB max dirty page
          0xfdc45908d60:leaf: 293 pages, 41112.37 KB, 92/201 clean/dirty pages, 12581.01 /28531.36 /2373.75 clean/dirty/updates KB, 222.51 KB max page, 222.51 KB max dirty page
          0xfdc45908d60:file:T00001.wt(<live>):
          0xfdc45908d60:internal: 98 pages, 617.01 KB, 22/76 clean/dirty pages, 133.36/483.65 clean / dirty KB, 6.89 KB max page, 6.89 KB max dirty page
          0xfdc45908d60:leaf: 398 pages, 47214.75 KB, 315/83 clean/dirty pages, 36819.43 /10395.31 /937.65 clean/dirty/updates KB, 196.26 KB max page, 126.74 KB max dirty page
          0xfdc45908d60:file:F00003(<live>):
          0xfdc45908d60:internal: 643 pages, 2582.30 KB, 487/156 clean/dirty pages, 1765.60/816.70 clean / dirty KB, 85.52 KB max page, 85.52 KB max dirty page
          0xfdc45908d60:leaf: 1504 pages, 59174.10 KB, 1396/108 clean/dirty pages, 54566.35 /4607.75 /278.62 clean/dirty/updates KB, 62.20 KB max page, 62.20 KB max dirty page
          0xfdc45908d60:file:WiredTigerHS.wt(<live>):
          0xfdc45908d60:internal: 1 pages, 0.56 KB, 0/1 clean/dirty pages, 0.00/0.56 clean / dirty KB, 0.56 KB max page, 0.56 KB max dirty page
          0xfdc45908d60:leaf: 1 pages, 22.31 KB, 0/1 clean/dirty pages, 0.00 /22.31 /2.19 clean/dirty/updates KB, 22.31 KB max page, 22.31 KB max dirty page
          0xfdc45908d60:file:WiredTiger.wt(<live>):
          0xfdc45908d60:internal: 1 pages, 0.61 KB, 1/0 clean/dirty pages, 0.61/0.00 clean / dirty KB, 0.61 KB max page, 0.00 KB max dirty page
          0xfdc45908d60:leaf: 1 pages, 19.78 KB, 1/0 clean/dirty pages, 19.78 /0.00 /12.79 clean/dirty/updates KB, 19.78 KB max page, 0.00 KB max dirty page
          0xfdc45908d60:cache dump: total found: 159.59 MB vs tracked inuse 148.95 MB
          0xfdc45908d60:total dirty bytes: 44.02 MB vs tracked dirty 45.20 MB
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_mongo_v6.1_ubuntu1804_ppc_format_stress_sanitizer_test_ppc_ubuntu_9f60092c26a08601c2728269d3a42cd8a49b7d1d_22_11_16_02_16_22/0?type=T#L5953
      
      Verbose Error message
          [1668567441:368639][23806:0x7ac84af6f180], t, eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_evict_thread_run, 324: cache eviction thread error: Connection timed out
          [1668567441:368654][23806:0x7ac84af6f180], t, eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_evict_thread_run, 324: the process must exit and restart: WT_PANIC: WiredTiger library panic
          [1668567441:368664][23806:0x7ac84af6f180], t, eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 28: aborting WiredTiger library
          [1668567443:789389][24379:0x7b204301f180], t, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_server, 473: Cache stuck for too long, giving up: Connection timed out
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_mongo_v6.1_ubuntu1804_ppc_format_stress_sanitizer_test_ppc_ubuntu_9f60092c26a08601c2728269d3a42cd8a49b7d1d_22_11_16_02_16_22/0?type=T#L6004
      
      Cache Stuck Failures
          0x10e83b9ba48:transaction state dump
          0x10e83b9ba48:current ID: 159092
          0x10e83b9ba48:last running ID: 159092
          0x10e83b9ba48:metadata_pinned ID: 144755
          0x10e83b9ba48:oldest ID: 159092
          0x10e83b9ba48:durable timestamp: (0, 106508)
          0x10e83b9ba48:oldest timestamp: (0, 106324)
          0x10e83b9ba48:pinned timestamp: (0, 106324)
          0x10e83b9ba48:stable timestamp: (0, 106324)
          0x10e83b9ba48:has_durable_timestamp: yes
          0x10e83b9ba48:has_oldest_timestamp: yes
          0x10e83b9ba48:has_pinned_timestamp: yes
          0x10e83b9ba48:has_stable_timestamp: yes
          0x10e83b9ba48:oldest_is_pinned: yes
          0x10e83b9ba48:stable_is_pinned: yes
          0x10e83b9ba48:checkpoint running: yes
          0x10e83b9ba48:checkpoint generation: 13
          0x10e83b9ba48:checkpoint pinned ID: 144649
          0x10e83b9ba48:checkpoint txn ID: 144755
          0x10e83b9ba48:session count: 33
          0x10e83b9ba48:Transaction state of active sessions:
          0x10e83b9ba48:ID: 0, pinned ID: 159092, metadata pinned ID: 144755, name: WT_CURSOR.search
          0x10e83b9ba48:transaction id: 0, mod count: 0, snap min: 144755, snap max: 159092, snapshot count: 1, commit_timestamp: (0, 0), durable_timestamp: (0, 106492), first_commit_timestamp: (0, 0), prepare_timestamp: (0, 0), pinned_durable_timestamp: (0, 0), read_timestamp: (0, 0), checkpoint LSN: [0][0], full checkpoint: false, rollback reason: , flags: 0x00001008, isolation: WT_ISO_SNAPSHOT
          0x10e83b9ba48:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
          0x10e83b9ba48:cache dump
          0x10e83b9ba48:cache full: no
          0x10e83b9ba48:cache clean check: no (72.450%)
          0x10e83b9ba48:cache dirty check: yes (20.001%)
          0x10e83b9ba48:cache updates check: no (3.132%)
          0x10e83b9ba48:file:T00002.wt(<live>):
          0x10e83b9ba48:internal: 17 pages, 13865.13 KB, 5/12 clean/dirty pages, 4560.64/9304.50 clean / dirty KB, 944.12 KB max page, 900.24 KB max dirty page
          0x10e83b9ba48:leaf: 59909 pages, 228255.45 KB, 42055/17854 clean/dirty pages, 149567.32 /78688.13 /12357.13 clean/dirty/updates KB, 254.37 KB max page, 254.37 KB max dirty page
          0x10e83b9ba48:file:T00001.wt(<live>):
          0x10e83b9ba48:internal: 1 pages, 38.82 KB, 0/1 clean/dirty pages, 0.00/38.82 clean / dirty KB, 38.82 KB max page, 38.82 KB max dirty page
          0x10e83b9ba48:leaf: 97 pages, 500.26 KB, 20/77 clean/dirty pages, 93.73 /406.54 /44.59 clean/dirty/updates KB, 5.67 KB max page, 5.67 KB max dirty page
          0x10e83b9ba48:file:T00003.wt(<live>):
          0x10e83b9ba48:internal: 44 pages, 11690.76 KB, 1/43 clean/dirty pages, 7.87/11682.88 clean / dirty KB, 318.09 KB max page, 318.09 KB max dirty page
          0x10e83b9ba48:leaf: 13250 pages, 32611.06 KB, 13250/0 clean/dirty pages, 32611.06 /0.00 /0.00 clean/dirty/updates KB, 4.98 KB max page, 0.00 KB max dirty page
          0x10e83b9ba48:file:WiredTigerHS.wt(<live>):
          0x10e83b9ba48:internal: 1 pages, 1.60 KB, 0/1 clean/dirty pages, 0.00/1.60 clean / dirty KB, 1.60 KB max page, 1.60 KB max dirty page
          0x10e83b9ba48:leaf: 0 pages
          0x10e83b9ba48:file:WiredTiger.wt(<live>):
          0x10e83b9ba48:internal: 1 pages, 0.61 KB, 1/0 clean/dirty pages, 0.61/0.00 clean / dirty KB, 0.61 KB max page, 0.00 KB max dirty page
          0x10e83b9ba48:leaf: 0 pages
          0x10e83b9ba48:cache dump: total found: 302.66 MB vs tracked inuse 280.41 MB
          0x10e83b9ba48:total dirty bytes: 97.78 MB vs tracked dirty 97.95 MB
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_mongo_v6.1_ubuntu1804_ppc_format_stress_sanitizer_test_ppc_ubuntu_9f60092c26a08601c2728269d3a42cd8a49b7d1d_22_11_16_02_16_22/0?type=T#L6179
      
      Verbose Error message
          [1668567443:843649][24379:0x7b204301f180], t, eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_evict_thread_run, 324: cache eviction thread error: Connection timed out
          [1668567443:843669][24379:0x7b204301f180], t, eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_evict_thread_run, 324: the process must exit and restart: WT_PANIC: WiredTiger library panic
          [1668567443:843681][24379:0x7b204301f180], t, eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 28: aborting WiredTiger library
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_mongo_v6.1_ubuntu1804_ppc_format_stress_sanitizer_test_ppc_ubuntu_9f60092c26a08601c2728269d3a42cd8a49b7d1d_22_11_16_02_16_22/0?type=T#L6226
      
      Failure keywords
      #4  0x00007b2043b23458 in __wt_panic_func (session=0x10e83b9ba48, error=<optimized out>, func=0x7b2043b91eb0 <__PRETTY_FUNCTION__.19426> "__wt_evict_thread_run", line=<optimized out>, category=<optimized out>, fmt=<optimized out>) at ../src/support/err.c:553
      #4  0x00007ac84cac3458 in __wt_panic_func (session=0xfdc45908d60, error=<optimized out>, func=0x7ac84cb31eb0 <__PRETTY_FUNCTION__.19426> "__wt_evict_thread_run", line=<optimized out>, category=<optimized out>, fmt=<optimized out>) at ../src/support/err.c:553
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_mongo_v6.1_ubuntu1804_ppc_format_stress_sanitizer_test_ppc_ubuntu_9f60092c26a08601c2728269d3a42cd8a49b7d1d_22_11_16_02_16_22/0?type=T#L6732
      

      logs

            Assignee:
            backlog-server-storage-engines [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Xgen-BuildBaron-User xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: