Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-3384

Test format stress on PPC aborts with stuck cache

    • Type: Icon: Task Task
    • Resolution: Gone away
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Labels:
      None

      Run
      http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-ppc/10813/

      Dump

      0x1000c720740:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
      0x1000c720740:transaction state dump
      0x1000c720740:current ID: 28634
      0x1000c720740:last running ID: 28634
      0x1000c720740:oldest ID: 26827
      0x1000c720740:oldest named snapshot ID: 0
      0x1000c720740:checkpoint running? no
      0x1000c720740:checkpoint generation: 1
      0x1000c720740:checkpoint pinned ID: 0
      0x1000c720740:checkpoint txn ID: 0
      0x1000c720740:session count: 56
      0x1000c720740:Transaction state of active sessions:
      0x1000c720740:ID:      0, mod count: 0, pinned ID: 28418, snap min: 28418, snap max: 28558, metadata pinned ID: 0, flags: 0x00000008, name: WT_CURSOR.search_near, isolation: WT_ISO_READ_COMMITTED
      0x1000c720740:ID:      0, mod count: 0, pinned ID: 28625, snap min: 28625, snap max: 28625, metadata pinned ID: 0, flags: 0x00000008, name: WT_CURSOR.search_near, isolation: WT_ISO_READ_COMMITTED
      0x1000c720740:ID:      0, mod count: 0, pinned ID: 28625, snap min: 28625, snap max: 28625, metadata pinned ID: 0, flags: 0x00000048, name: WT_CURSOR.search, isolation: WT_ISO_READ_COMMITTED
      0x1000c720740:ID:      0, mod count: 0, pinned ID: 28625, snap min: 28625, snap max: 28628, metadata pinned ID: 0, flags: 0x00000008, name: WT_CURSOR.search, isolation: WT_ISO_READ_COMMITTED
      0x1000c720740:ID:      0, mod count: 0, pinned ID: 28621, snap min: 28621, snap max: 28625, metadata pinned ID: 0, flags: 0x00000048, name: WT_CURSOR.search, isolation: WT_ISO_READ_COMMITTED
      0x1000c720740:ID:      0, mod count: 0, pinned ID: 28631, snap min: 28631, snap max: 28631, metadata pinned ID: 0, flags: 0x00000008, name: WT_CURSOR.search, isolation: WT_ISO_READ_COMMITTED
      0x1000c720740:ID:      0, mod count: 0, pinned ID: 28625, snap min: 28625, snap max: 28628, metadata pinned ID: 0, flags: 0x00000008, name: WT_CURSOR.search, isolation: WT_ISO_READ_COMMITTED
      0x1000c720740:ID:      0, mod count: 0, pinned ID: 26827, snap min: 26827, snap max: 28546, metadata pinned ID: 0, flags: 0x00000008, name: WT_CURSOR.search_near, isolation: WT_ISO_READ_COMMITTED
      0x1000c720740:ID:      0, mod count: 0, pinned ID: 26827, snap min: 26827, snap max: 28554, metadata pinned ID: 0, flags: 0x00000048, name: WT_CURSOR.search_near, isolation: WT_ISO_READ_COMMITTED
      0x1000c720740:ID:      0, mod count: 0, pinned ID: 28625, snap min: 28625, snap max: 28625, metadata pinned ID: 0, flags: 0x00000048, name: WT_CURSOR.search, isolation: WT_ISO_READ_COMMITTED
      0x1000c720740:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
      0x1000c720740:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
      0x1000c720740:cache dump
      0x1000c720740:file:wt-000011.bf(<live>):
      0x1000c720740:internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:file:wt-000011.lsm(checkpoint=WiredTigerCheckpoint.1):
      0x1000c720740:internal: 3 pages, 0MB, 3/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:leaf: 2 pages, 0MB, 2/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:file:wt-000010.bf(<live>):
      0x1000c720740:internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:file:wt-000010.lsm(checkpoint=WiredTigerCheckpoint.1):
      0x1000c720740:internal: 3 pages, 0MB, 3/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:leaf: 2 pages, 0MB, 2/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:file:wt-000009.bf(<live>):
      0x1000c720740:internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:file:wt-000008.bf(<live>):
      0x1000c720740:internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:file:wt-000007.bf(<live>):
      0x1000c720740:internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:file:wt-000009.lsm(checkpoint=WiredTigerCheckpoint.1):
      0x1000c720740:internal: 4 pages, 0MB, 4/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:leaf: 2 pages, 0MB, 2/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:file:wt-000008.lsm(checkpoint=WiredTigerCheckpoint.1):
      0x1000c720740:internal: 3 pages, 0MB, 3/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:leaf: 3 pages, 0MB, 3/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:file:wt-000007.lsm(checkpoint=WiredTigerCheckpoint.1):
      0x1000c720740:internal: 4 pages, 0MB, 4/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:leaf: 2 pages, 0MB, 2/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:file:wt-000006.bf(<live>):
      0x1000c720740:internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:file:wt-000005.bf(<live>):
      0x1000c720740:internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:file:wt-000015.lsm(<live>):
      0x1000c720740:internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:leaf: 1 pages, 0MB, 0/1 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:file:wt-000006.lsm(checkpoint=WiredTigerCheckpoint.1):
      0x1000c720740:internal: 2 pages, 0MB, 2/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:leaf: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:file:wt-000005.lsm(checkpoint=WiredTigerCheckpoint.1):
      0x1000c720740:internal: 3 pages, 0MB, 3/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:leaf: 4 pages, 0MB, 4/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:file:wt-000014.lsm(<live>):
      0x1000c720740:internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:leaf: 1 pages, 0MB, 0/1 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:file:wt-000004.bf(<live>):
      0x1000c720740:internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:file:wt-000013.lsm(<live>):
      0x1000c720740:internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:leaf: 1 pages, 2MB, 0/1 clean/dirty pages, 0/2 clean/dirty MB, 2MB max page, 2MB max dirty page
      0x1000c720740:file:wt-000004.lsm(checkpoint=WiredTigerCheckpoint.1):
      0x1000c720740:internal: 7 pages, 0MB, 7/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:leaf: 7 pages, 0MB, 7/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:file:wt-000012.lsm(<live>):
      0x1000c720740:internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:leaf: 1 pages, 2MB, 0/1 clean/dirty pages, 0/2 clean/dirty MB, 2MB max page, 2MB max dirty page
      0x1000c720740:file:wt-000011.lsm(<live>):
      0x1000c720740:internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:leaf: 1 pages, 2MB, 1/0 clean/dirty pages, 2/0 clean/dirty MB, 2MB max page, 0MB max dirty page
      0x1000c720740:file:wt-000003.bf(<live>):
      0x1000c720740:internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:file:wt-000003.lsm(checkpoint=WiredTigerCheckpoint.1):
      0x1000c720740:internal: 4 pages, 0MB, 4/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:leaf: 7 pages, 0MB, 7/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:file:wt-000010.lsm(<live>):
      0x1000c720740:internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:leaf: 1 pages, 3MB, 1/0 clean/dirty pages, 3/0 clean/dirty MB, 3MB max page, 0MB max dirty page
      0x1000c720740:file:wt-000009.lsm(<live>):
      0x1000c720740:internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:leaf: 1 pages, 2MB, 1/0 clean/dirty pages, 2/0 clean/dirty MB, 2MB max page, 0MB max dirty page
      0x1000c720740:file:wt-000008.lsm(<live>):
      0x1000c720740:internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:leaf: 1 pages, 3MB, 1/0 clean/dirty pages, 3/0 clean/dirty MB, 3MB max page, 0MB max dirty page
      0x1000c720740:file:wt-000007.lsm(<live>):
      0x1000c720740:internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:leaf: 1 pages, 3MB, 1/0 clean/dirty pages, 3/0 clean/dirty MB, 3MB max page, 0MB max dirty page
      0x1000c720740:file:wt-000006.lsm(<live>):
      0x1000c720740:internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:leaf: 1 pages, 2MB, 1/0 clean/dirty pages, 2/0 clean/dirty MB, 2MB max page, 0MB max dirty page
      0x1000c720740:file:wt-000005.lsm(<live>):
      0x1000c720740:internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:leaf: 1 pages, 3MB, 1/0 clean/dirty pages, 3/0 clean/dirty MB, 3MB max page, 0MB max dirty page
      0x1000c720740:file:wt-000002.lsm(checkpoint=WiredTigerCheckpoint.1):
      0x1000c720740:internal: 17 pages, 0MB, 17/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:leaf: 7 pages, 0MB, 7/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:file:WiredTigerLAS.wt(<live>):
      0x1000c720740:internal: 1 pages, 0MB, 0/1 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:file:WiredTiger.wt(<live>):
      0x1000c720740:internal: 1 pages, 0MB, 0/1 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x1000c720740:cache dump: total found: 28MB vs tracked inuse 28MB
      0x1000c720740:total dirty bytes: 4MB
      0x1000c720740:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
      [1498137845:387216][19980:0x3fffadb9f1b0], t, eviction-server: cache eviction thread error: Connection timed out
      [1498137845:387225][19980:0x3fffadb9f1b0], t, eviction-server: the process must exit and restart: WT_PANIC: WiredTiger library panic
      [1498137845:387231][19980:0x3fffadb9f1b0], t, eviction-server: aborting WiredTiger library
      
      *** Aborted
      Register dump:
      sr0=00003fffadede4d8 sr1=800000010280d033 dar=00003fff74151188 dsi=40000000
      lr=00003fffadee0658  ctr=0000000000000000 gr3*=0000000000004e0c trap=00000c00
      ccr=44002824  xer=00000000
      gr0-3:   00000000000000fa 00003fffadb9e440 00003fffae079768 0000000000000000
      gr4-7:   0000000000004e25 0000000000000006 0000000000000008 0000000000000000
      gr8-11:  0000000000000000 0000000000000000 0000000000000000 0000000000000000
      gr12-15: 0000000000000000 00003fffadba68e0 0000000000000000 0000000000000000
      gr16-19: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
      gr20-23: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
      gr24-27: 00000000103c0fd8 00000000103c0fd8 00000000103c0fd8 00000000103c0fd8
      gr28-31: 00003fffae1643b8 00000000003d0f00 00003fffadb9e4f8 0000000000000006
      fscr=0000000000000000
      fp0-3:   00000000b220005f 41d652f0e3c00000 0000000000000000 0000000000000000
      fp4-7:   0000000000000000 0000000000000000 0000000000000000 0000000000000000
      fp8-11:  00000000deadbeef 00000000deadbeef 41d652f0e3c00000 7ff0000000000000
      fp12-15: 417e000010000000 0000000000000000 0000000000000000 0000000000000000
      fp16-19: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
      fp20-23: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
      fp24-27: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
      fp28-31: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
      
      Backtrace:
      /lib64/libSegFault.so(+0x1734)[0x3fffae1e1734]
      ??:0(??)[0x3fffae210478]
      /lib64/power8/libc.so.6(abort+0x27c)[0x3fffadee072c]
      /home/jenkins/jenkins/workspace/wiredtiger-test-format-stress-ppc/build_posix/../src/os_common/os_abort.c:25(__wt_abort)[0x100697d0]
      /home/jenkins/jenkins/workspace/wiredtiger-test-format-stress-ppc/build_posix/../src/support/err.c:503(__wt_panic)[0x100b7718]
      /home/jenkins/jenkins/workspace/wiredtiger-test-format-stress-ppc/build_posix/../src/evict/evict_lru.c:319(__wt_evict_thread_run)[0x100405dc]
      /home/jenkins/jenkins/workspace/wiredtiger-test-format-stress-ppc/build_posix/../src/support/thread_group.c:31(__thread_run)[0x100c63a4]
      /lib64/power8/libpthread.so.0(+0x8944)[0x3fffae138944]
      /lib64/power8/libc.so.6(clone+0x98)[0x3fffadfb7640]
      

      Config

      ############################################
      #  RUN PARAMETERS
      ############################################
      abort=0
      alter=0
      auto_throttle=1
      backups=0
      bitcnt=5
      bloom=1
      bloom_bit_count=37
      bloom_hash_count=27
      bloom_oldest=0
      cache=30
      checkpoints=1
      checksum=uncompressed
      chunk_size=1
      compaction=0
      compression=none
      data_extend=0
      data_source=lsm
      delete_pct=19
      dictionary=0
      direct_io=0
      encryption=none
      evict_max=1
      file_type=row-store
      firstfit=0
      huffman_key=0
      huffman_value=0
      in_memory=0
      insert_pct=66
      internal_key_truncation=1
      internal_page_max=9
      isolation=read-committed
      key_gap=16
      key_max=256
      key_min=256
      leaf_page_max=9
      leak_memory=0
      logging=0
      logging_archive=0
      logging_compression=none
      logging_prealloc=0
      long_running_txn=0
      lsm_worker_threads=4
      merge_max=4
      mmap=1
      modify_pct=4
      ops=100000
      prefix_compression=1
      prefix_compression_min=2
      quiet=1
      read_pct=3
      rebalance=1
      repeat_data_pct=54
      reverse=0
      rows=100000
      runs=1
      salvage=1
      split_pct=76
      statistics=0
      statistics_server=0
      threads=30
      timer=20
      transaction-frequency=34
      value_max=1544
      value_min=256
      verify=1
      wiredtiger_config=
      write_pct=8
      ############################################
      

            Assignee:
            alexander.gorrod@mongodb.com Alexander Gorrod
            Reporter:
            david.hows David Hows
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: