Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-3645

Build Failed: Lookaside file occupies 10MB of 11MB cache size

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 3.6.0-rc0, WT3.0.0
    • Affects Version/s: None
    • Component/s: None
    • Labels:
      None
    • Storage 2017-11-13

      Following build failure is seen :
      http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer-old-branches/560/

      The test config is :

      abort=0
      alter=0
      auto_throttle=1
      backups=0
      bitcnt=7
      bloom=1
      bloom_bit_count=62
      bloom_hash_count=28
      bloom_oldest=0
      cache=64
      checkpoints=1
      checksum=uncompressed
      chunk_size=8
      compaction=0
      compression=lz4-noraw
      data_extend=0
      data_source=table
      delete_pct=20
      dictionary=0
      direct_io=0
      encryption=none
      evict_max=4
      file_type=row-store
      firstfit=1
      huffman_key=0
      huffman_value=0
      independent_thread_rng=1
      in_memory=0
      insert_pct=12
      internal_key_truncation=1
      internal_page_max=11
      isolation=random
      key_gap=17
      key_max=125
      key_min=11
      leaf_page_max=17
      leak_memory=0
      logging=0
      logging_archive=0
      logging_compression=none
      logging_prealloc=1
      long_running_txn=0
      lsm_worker_threads=4
      merge_max=14
      mmap=1
      modify_pct=48
      ops=100000
      prefix_compression=0
      prefix_compression_min=5
      quiet=1
      read_pct=13
      rebalance=1
      repeat_data_pct=19
      reverse=0
      rows=100000
      runs=1
      salvage=1
      split_pct=60
      statistics=1
      statistics_server=0
      threads=9
      timer=360
      transaction_timestamps=1
      transaction-frequency=11
      value_max=166
      value_min=0
      verify=1
      wiredtiger_config=
      write_pct=7
      

      Cache is stuck, and here is the cache dump:

      0x634000001700:file:wt.wt(<live>):
      0x634000001700:internal: 1 pages, 0MB, 0/1 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x634000001700:leaf: 1 pages, 0MB, 0/1 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x634000001700:file:WiredTigerLAS.wt(<live>):
      0x634000001700:internal: 1 pages, 0MB, 0/1 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x634000001700:leaf: 2 pages, 10MB, 1/1 clean/dirty pages, 0/10 clean/dirty MB, 10MB max page, 10MB max dirty page
      0x634000001700:file:WiredTiger.wt(<live>):
      0x634000001700:internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x634000001700:leaf: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x634000001700:cache dump: total found: 11MB vs tracked inuse 14MB
      0x634000001700:total dirty bytes: 11MB
      [1507626690:605168][8845:0x7f5d9bff9700], t, eviction-server: cache eviction thread error: Connection timed out
      [1507626690:605178][8845:0x7f5d9bff9700], t, eviction-server: the process must exit and restart: WT_PANIC: WiredTiger library panic
      [1507626690:605186][8845:0x7f5d9bff9700], t, eviction-server: aborting WiredTiger library
      

      The cache is full of leaf pages from a lookaside file:

      0x634000001700:file:WiredTigerLAS.wt(<live>):
      ...
      0x634000001700:leaf: 2 pages, 10MB, 1/1 clean/dirty pages, 0/10 clean/dirty MB, 10MB max page, 10MB max dirty page
      ...
      0x634000001700:total dirty bytes: 11MB
      

            Assignee:
            michael.cahill@mongodb.com Michael Cahill (Inactive)
            Reporter:
            neha.khatri Neha Khatri
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: