Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-5276

test/format needs to move the oldest timestamp forward during bulk load

    • 8
    • Storage Engines 2019-12-02, Storage Engines 2019-12-16

      This is another cache stuck clean case after the recent merge of WT-5082, on ocelot-aws. In this specific case, the cache was full, no timestamp involved, and there's a core dumped. 

      http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/25332/

      ++ nice ./t -1 -c ../../../test/format/CONFIG.stress file_type=var leaf_page_max=9 internal_page_max=9 value_min=256
      t: process 31902 running
      [1575053832:616423][31902:0x7f503f6f9700], t, eviction-server: __evict_server, 440: Cache stuck for too long, giving up: Connection timed out
      0x7f5040037e80:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
      0x7f5040037e80:transaction state dump
      0x7f5040037e80:current ID: 3
      0x7f5040037e80:last running ID: 3
      0x7f5040037e80:metadata_pinned ID: 3
      0x7f5040037e80:oldest ID: 3
      0x7f5040037e80:durable timestamp: (0,0)
      0x7f5040037e80:oldest timestamp: (0,0)
      0x7f5040037e80:pinned timestamp: (0,0)
      0x7f5040037e80:stable timestamp: (0,0)
      0x7f5040037e80:has_durable_timestamp: no
      0x7f5040037e80:has_oldest_timestamp: no
      0x7f5040037e80:has_pinned_timestamp: no
      0x7f5040037e80:has_stable_timestamp: no
      0x7f5040037e80:oldest_is_pinned: no
      0x7f5040037e80:stable_is_pinned: no
      0x7f5040037e80:checkpoint running: no
      0x7f5040037e80:checkpoint generation: 1
      0x7f5040037e80:checkpoint pinned ID: 0
      0x7f5040037e80:checkpoint txn ID: 0
      0x7f5040037e80:oldest named snapshot ID: 0
      0x7f5040037e80:session count: 11
      0x7f5040037e80:Transaction state of active sessions:
      0x7f5040037e80:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
      0x7f5040037e80:cache dump
      0x7f5040037e80:cache full: yes
      0x7f5040037e80:cache clean check: yes (115.378%)
      0x7f5040037e80:cache dirty check: no (0.003%)
      0x7f5040037e80:file:wt.wt(<live>) eviction disabled at open:
      0x7f5040037e80:  Opened exclusively. Cannot walk tree, skipping.
      0x7f5040037e80:file:WiredTigerLAS.wt(<live>) eviction disabled at open:
      0x7f5040037e80:internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x7f5040037e80:leaf: 0 pages
      0x7f5040037e80:file:WiredTiger.wt(<live>):
      0x7f5040037e80:internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x7f5040037e80:leaf: 1 pages, 0MB, 0/1 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x7f5040037e80:cache dump: total found: 0MB vs tracked inuse 166MB
      0x7f5040037e80:total dirty bytes: 0MB
      [1575053832:617658][31902:0x7f503f6f9700], t, eviction-server: __wt_evict_thread_run, 315: cache eviction thread error: Connection timed out
      [1575053832:617743][31902:0x7f503f6f9700], t, eviction-server: __wt_panic, 490: the process must exit and restart: WT_PANIC: WiredTiger library panic
      [1575053832:617819][31902:0x7f503f6f9700], t, eviction-server: __wt_abort, 28: aborting WiredTiger library
      /tmp/jenkins18201607046064919540.sh: line 28: 31902 Aborted                 (core dumped) nice ./t -1 -c ../../../test/format/CONFIG.stress file_type=var leaf_page_max=9 internal_page_max=9 value_min=256 

      The configuration:

      ############################################
      #  RUN PARAMETERS
      ############################################
      abort=0
      alter=0
      assert_commit_timestamp=0
      assert_read_timestamp=0
      auto_throttle=1
      backups=0
      bitcnt=1
      bloom=1
      bloom_bit_count=54
      bloom_hash_count=16
      bloom_oldest=0
      cache=144
      cache_minimum=20
      checkpoints=on
      checkpoint_log_size=24
      checkpoint_wait=80
      checksum=uncompressed
      chunk_size=5
      compaction=0
      compression=zlib
      data_extend=0
      data_source=table
      delete_pct=1
      dictionary=0
      direct_io=0
      encryption=none
      evict_max=1
      file_type=variable-length column-store
      firstfit=0
      huffman_key=0
      huffman_value=0
      independent_thread_rng=0
      in_memory=0
      insert_pct=5
      internal_key_truncation=1
      internal_page_max=9
      isolation=snapshot
      key_gap=11
      key_max=78
      key_min=28
      leaf_page_max=9
      leak_memory=0
      logging=0
      logging_archive=1
      logging_compression=none
      logging_file_max=342005
      logging_prealloc=0
      long_running_txn=0
      lsm_worker_threads=3
      memory_page_max=9
      merge_max=17
      mmap=1
      modify_pct=92
      ops=0
      prefix_compression=1
      prefix_compression_min=3
      prepare=0
      quiet=1
      random_cursor=0
      read_pct=1
      rebalance=1
      repeat_data_pct=7
      reverse=0
      rows=1000000
      runs=1
      salvage=1
      split_pct=63
      statistics=0
      statistics_server=0
      threads=8
      timer=4
      timing_stress_aggressive_sweep=0
      timing_stress_checkpoint=0
      timing_stress_lookaside_sweep=0
      timing_stress_split_1=0
      timing_stress_split_2=0
      timing_stress_split_3=0
      timing_stress_split_4=0
      timing_stress_split_5=0
      timing_stress_split_6=0
      timing_stress_split_7=0
      timing_stress_split_8=0
      transaction_timestamps=1
      transaction-frequency=100
      truncate=1
      value_max=827
      value_min=256
      verify=1
      wiredtiger_config=
      write_pct=1
      ############################################ 

            Assignee:
            keith.bostic@mongodb.com Keith Bostic (Inactive)
            Reporter:
            luke.chen@mongodb.com Luke Chen
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: