Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-7010

Fix cache stuck with updates in cache failure

    • Type: Icon: Build Failure Build Failure
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Labels:
    • 8
    • Storage - Ra 2021-07-26, Storage - Ra 2021-11-01
    • v5.1

      format-stress-sanitizer-ppc-test failed on ~ Ubuntu 18.04 PPC

      Host: ubuntu1804-ppc-5.pic.build.10gen.cc
      Project: WiredTiger (develop)
      Commit: diff: WT-6831 Retry search if we race with prepare update commit/rollback (#6166)

      Irrespective of whether the prepared update is visible or not, retry
      the search again when the prepared update is either committed/rollbacked
      in-parallel to search. | 08 Dec 20 22:38 UTC
      Evergreen Subscription: ; Evergreen Event:

      [Task Logs

      https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu1804_ppc_format_stress_sanitizer_ppc_test_31c4748b76996ebfe3c5e8d233b457719f0f8d28_20_12_08_22_38_05/0?type=T]

      Symptom:

      [2020/12/09 00:35:19.588]     t: process 18689 running
       [2020/12/09 00:35:19.588]     0xba000023680:oldest pinned transaction ID rolled back for eviction
       [2020/12/09 00:35:19.588]     0xba000024dc0:oldest pinned transaction ID rolled back for eviction
       [2020/12/09 00:35:19.588]     0xba000024600:oldest pinned transaction ID rolled back for eviction
       [2020/12/09 00:35:19.588]     0xba0000251a0:oldest pinned transaction ID rolled back for eviction
       [2020/12/09 00:35:19.588]     [1607473059:438621][18689:0x756ae16cf100], eviction-server: __evict_server, 446: Cache stuck for too long, giving up: Connection timed out
       [2020/12/09 00:35:19.588]     0xba000021f40:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
       [2020/12/09 00:35:19.588]     0xba000021f40:transaction state dump
       [2020/12/09 00:35:19.588]     0xba000021f40:current ID: 29006
       [2020/12/09 00:35:19.588]     0xba000021f40:last running ID: 29006
       [2020/12/09 00:35:19.588]     0xba000021f40:metadata_pinned ID: 27637
       [2020/12/09 00:35:19.588]     0xba000021f40:oldest ID: 29006
       [2020/12/09 00:35:19.588]     0xba000021f40:durable timestamp: (0, 46820)
       [2020/12/09 00:35:19.588]     0xba000021f40:oldest timestamp: (0, 46819)
       [2020/12/09 00:35:19.588]     0xba000021f40:pinned timestamp: (0, 46819)
       [2020/12/09 00:35:19.588]     0xba000021f40:stable timestamp: (0, 46820)
       [2020/12/09 00:35:19.588]     0xba000021f40:has_durable_timestamp: yes
       [2020/12/09 00:35:19.588]     0xba000021f40:has_oldest_timestamp: yes
       [2020/12/09 00:35:19.588]     0xba000021f40:has_pinned_timestamp: yes
       [2020/12/09 00:35:19.588]     0xba000021f40:has_stable_timestamp: yes
       [2020/12/09 00:35:19.588]     0xba000021f40:oldest_is_pinned: yes
       [2020/12/09 00:35:19.588]     0xba000021f40:stable_is_pinned: no
       [2020/12/09 00:35:19.588]     0xba000021f40:checkpoint running: yes
       [2020/12/09 00:35:19.588]     0xba000021f40:checkpoint generation: 71
       [2020/12/09 00:35:19.588]     0xba000021f40:checkpoint pinned ID: 27637
       [2020/12/09 00:35:19.588]     0xba000021f40:checkpoint txn ID: 27637
       [2020/12/09 00:35:19.588]     0xba000021f40:session count: 24
       [2020/12/09 00:35:19.588]     0xba000021f40:Transaction state of active sessions:
       [2020/12/09 00:35:19.588]     0xba000021f40:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
       [2020/12/09 00:35:19.588]     0xba000021f40:cache dump
       [2020/12/09 00:35:19.588]     0xba000021f40:cache full: no
       [2020/12/09 00:35:19.588]     0xba000021f40:cache clean check: no (61.989%)
       [2020/12/09 00:35:19.588]     0xba000021f40:cache dirty check: no (8.200%)
       [2020/12/09 00:35:19.588]     0xba000021f40:cache updates check: yes (10.658%)
       [2020/12/09 00:35:19.588]     0xba000021f40:file:wt(<live>):
       [2020/12/09 00:35:19.588]     0xba000021f40:internal: 3445 pages, 15MB, 2706/739 clean/dirty pages, 12/3 clean/dirty MB, 0MB max page, 0MB max dirty page
       [2020/12/09 00:35:19.588]     0xba000021f40:leaf: 55221 pages, 85MB, 48746/6475 clean/dirty pages, 72/13/17 clean/dirty/updates MB, 0MB max page, 0MB max dirty page
       [2020/12/09 00:35:19.588]     0xba000021f40:file:WiredTigerHS.wt(<live>):
       [2020/12/09 00:35:19.588]     0xba000021f40:internal: 1 pages, 0MB, 0/1 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
       [2020/12/09 00:35:19.588]     0xba000021f40:leaf: 0 pages
       [2020/12/09 00:35:19.588]     0xba000021f40:file:WiredTiger.wt(<live>):
       [2020/12/09 00:35:19.588]     0xba000021f40:internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
       [2020/12/09 00:35:19.588]     0xba000021f40:leaf: 0 pages
       [2020/12/09 00:35:19.588]     0xba000021f40:cache dump: total found: 109MB vs tracked inuse 101MB
       [2020/12/09 00:35:19.588]     0xba000021f40:total dirty bytes: 16MB vs tracked dirty 16MB
       [2020/12/09 00:35:19.588]     0xba000021f40:total updates bytes: 17MB vs tracked updates 18MB
       [2020/12/09 00:35:19.588]     [1607473059:542376][18689:0x756ae16cf100], eviction-server: __wt_evict_thread_run, 321: cache eviction thread error: Connection timed out
       [2020/12/09 00:35:19.588]     [1607473059:542449][18689:0x756ae16cf100], eviction-server: __wt_evict_thread_run, 321: the process must exit and restart: WT_PANIC: WiredTiger library panic
       [2020/12/09 00:35:19.588]     [1607473059:542503][18689:0x756ae16cf100], eviction-server: __wt_abort, 28: aborting WiredTiger library
       [2020/12/09 00:35:19.590] format.sh: /data/mci/c06990f54b16f4f7570965b69763bce6/wiredtiger/test/format/RUNDIR.89/CONFIG: 

            Assignee:
            jie.chen@mongodb.com Jie Chen
            Reporter:
            xgen-evg-user Xgen-Evergreen-User
            Alexey Anisimov (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            13 Start watching this issue

              Created:
              Updated:
              Resolved: