Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-5082

test/format stress job hung full of clean pages on zSeries

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major - P3
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 4.3 Desired
    • Component/s: None

      Description

      There's a hung of the test/format stress job on zSeries machine.

      http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-zseries/49467/console

      Used gdb generated the core, and here is the stack:

      (gdb) bt
      #0  0x000003ffa290c566 in __pthread_cond_timedwait (cond=0x93098200, mutex=0x930981d8, abstime=0x3ffa21fec68)
          at pthread_cond_timedwait.c:166
      #1  0x00000000800954e4 in __wt_cond_wait_signal (session=0x3ffa2bcb590, cond=0x930981d0, usecs=10000,
          run_func=0x0, signalled=0x3ffa21fed7f) at ../src/os_posix/os_mtx_cond.c:121
      #2  0x00000000800626a8 in __wt_cond_wait (session=0x3ffa2bcb590, cond=0x930981d0, usecs=10000, run_func=0x0)
          at ../src/include/misc.i:19
      #3  0x0000000080067c6e in __evict_lru_pages (session=0x3ffa2bcb590, is_server=false)
          at ../src/evict/evict_lru.c:1126
      #4  0x0000000080065970 in __wt_evict_thread_run (session=0x3ffa2bcb590, thread=0x930982b0)
          at ../src/evict/evict_lru.c:314
      #5  0x00000000800f7bb2 in __thread_run (arg=0x930982b0) at ../src/support/thread_group.c:31
      #6  0x000003ffa29081f2 in start_thread (arg=0x3ffa21ff910) at pthread_create.c:310
      #7  0x000003ffa26898da in thread_start () at ../sysdeps/unix/sysv/linux/s390/s390-64/clone.S:76 

      Cache is full of clean page:

      0x3ffa2bcc3a0:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
      0x3ffa2bcc3a0:cache dump
      0x3ffa2bcc3a0:cache full: yes
      0x3ffa2bcc3a0:cache clean check: yes (114.145%)
      0x3ffa2bcc3a0:cache dirty check: no (2.281%)
      0x3ffa2bcc3a0:file:wt(<live>):
      0x3ffa2bcc3a0:internal: 680 pages, 81MB, 667/13 clean/dirty pages, 76/4 clean/dirty MB, 2MB max page, 2MB max dirty page
      0x3ffa2bcc3a0:leaf: 0 pages
      0x3ffa2bcc3a0:file:WiredTigerLAS.wt(<live>):
      0x3ffa2bcc3a0:internal: 3 pages, 2MB, 1/2 clean/dirty pages, 0/2 clean/dirty MB, 2MB max page, 2MB max dirty page
      0x3ffa2bcc3a0:leaf: 2 pages, 1MB, 0/2 clean/dirty pages, 0/1 clean/dirty MB, 1MB max page, 1MB max dirty page
      0x3ffa2bcc3a0:file:WiredTiger.wt(<live>):
      0x3ffa2bcc3a0:internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x3ffa2bcc3a0:leaf: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x3ffa2bcc3a0:cache dump: total found: 92MB vs tracked inuse 92MB
      0x3ffa2bcc3a0:total dirty bytes: 8MB
      [1566954009:134326][25525:0x3ffa09ff910], t, eviction-server: __wt_evict_thread_run, 318: cache eviction thread error: Connection timed out
      [1566954009:134351][25525:0x3ffa09ff910], t, eviction-server: __wt_panic, 494: the process must exit and restart: WT_PANIC: WiredTiger library panic
      [1566954009:134374][25525:0x3ffa09ff910], t, eviction-server: __wt_abort, 28: aborting WiredTiger library
      *** Aborted 

      The configuration:

      ############################################
      #  RUN PARAMETERS
      ############################################
      abort=0
      alter=0
      assert_commit_timestamp=0
      assert_read_timestamp=0
      auto_throttle=1
      backups=0
      bitcnt=6
      bloom=1
      bloom_bit_count=28
      bloom_hash_count=20
      bloom_oldest=0
      cache=81
      cache_minimum=20
      checkpoints=wiredtiger
      checkpoint_log_size=119
      checkpoint_wait=7
      checksum=uncompressed
      chunk_size=10
      compaction=0
      compression=snappy
      data_extend=0
      data_source=file
      delete_pct=13
      dictionary=0
      direct_io=0
      encryption=none
      evict_max=0
      file_type=row-store
      firstfit=0
      huffman_key=0
      huffman_value=0
      independent_thread_rng=1
      in_memory=0
      insert_pct=9
      internal_key_truncation=1
      internal_page_max=9
      isolation=snapshot
      key_gap=2
      key_max=256
      key_min=256
      leaf_page_max=9
      leak_memory=0
      logging=0
      logging_archive=1
      logging_compression=none
      logging_file_max=295668
      logging_prealloc=0
      long_running_txn=0
      lsm_worker_threads=4
      memory_page_max=2
      merge_max=12
      mmap=1
      modify_pct=22
      ops=0
      prefix_compression=1
      prefix_compression_min=5
      prepare=0
      quiet=1
      read_pct=50
      rebalance=1
      repeat_data_pct=85
      reverse=1
      rows=1000000
      runs=1
      salvage=1
      split_pct=94
      statistics=0
      statistics_server=0
      threads=20
      timer=4
      timing_stress_aggressive_sweep=0
      timing_stress_checkpoint=0
      timing_stress_lookaside_sweep=0
      timing_stress_split_1=0
      timing_stress_split_2=0
      timing_stress_split_3=0
      timing_stress_split_4=0
      timing_stress_split_5=0
      timing_stress_split_6=0
      timing_stress_split_7=0
      timing_stress_split_8=0
      transaction_timestamps=1
      transaction-frequency=100
      truncate=1
      value_max=2385
      value_min=15
      verify=1
      wiredtiger_config=
      write_pct=6
      ############################################ 

        Attachments

          Activity

            People

            • Assignee:
              backlog-server-storage-engines Backlog - Storage Engines Team
              Reporter:
              luke.chen Luke Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: