Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-2953

Test format failure - stuck cache with 1/42MB dirty

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: WT2.9.0, 3.2.11, 3.4.0-rc3
    • Labels:
      None
    • # Replies:
      11
    • Last comment by Customer:
      true
    • Sprint:
      Storage 2016-11-21

      Description

      Run was on zSeries and failed with the following error on the tip of develop (6775c3401a4e51aeff02c0f369ba7a8a0b9af633)

      [1475808552:167916][37779:0x3ff877ff910], t, eviction-server: Cache stuck for too long, giving up: Connection timed out
      ==========
      cache dump
      file:wt.wt(<live>): 
      	internal: 1 pages, 0MB, 0/1 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      	leaf: 20 pages, 2MB, 14/6 clean/dirty pages, 0/1 clean/dirty MB, 0MB max page, 0MB max dirty page
      file:WiredTigerLAS.wt(<live>): 
      	internal: 1 pages, 0MB, 0/1 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      	leaf: 7 pages, 0MB, 7/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      file:WiredTiger.wt(<live>): 
      	internal: 1 pages, 0MB, 0/1 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      	leaf: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      cache dump: total found = 2MB vs tracked inuse 5MB
      total dirty bytes = 1MB
      ==========
      [1475808552:168023][37779:0x3ff877ff910], t, eviction-server: cache eviction thread error: Connection timed out
      [1475808552:168026][37779:0x3ff877ff910], t, eviction-server: the process must exit and restart: WT_PANIC: WiredTiger library panic
      [1475808552:168028][37779:0x3ff877ff910], t, eviction-server: aborting WiredTiger library
      

      Config:

      ############################################
      #  RUN PARAMETERS
      ############################################
      abort=0
      auto_throttle=1
      backups=1
      bitcnt=5
      bloom=1
      bloom_bit_count=34
      bloom_hash_count=15
      bloom_oldest=0
      cache=42
      checkpoints=1
      checksum=uncompressed
      chunk_size=4
      compaction=0
      compression=zlib
      data_extend=0
      data_source=table
      delete_pct=31
      dictionary=0
      direct_io=0
      encryption=none
      evict_max=2
      file_type=variable-length column-store
      firstfit=0
      huffman_key=0
      huffman_value=0
      in_memory=0
      insert_pct=25
      internal_key_truncation=1
      internal_page_max=16
      isolation=read-uncommitted
      key_gap=0
      key_max=99
      key_min=22
      leaf_page_max=15
      leak_memory=0
      logging=0
      logging_archive=1
      logging_compression=none
      logging_prealloc=0
      long_running_txn=0
      lsm_worker_threads=3
      merge_max=18
      mmap=1
      ops=100000
      prefix_compression=0
      prefix_compression_min=3
      quiet=1
      repeat_data_pct=29
      reverse=0
      rows=100000
      runs=100
      rebalance=1
      salvage=1
      split_pct=41
      statistics=0
      statistics_server=0
      threads=10
      timer=20
      transaction-frequency=9
      value_max=2641
      value_min=19
      verify=1
      wiredtiger_config=
      write_pct=60
      

      Run:
      http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-zseries/11362/consoleFull

      Michael attempted reproduction, but the issue did not reproduce readily on x86.

        Activity

        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'}

        Message: WT-2953 Only test for checkpoint/LAS collision if LAS was used. (#3111)
        Branch: develop
        https://github.com/wiredtiger/wiredtiger/commit/04cdcbdafd529588d8a4a80ecc308ad01d507a55

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'} Message: WT-2953 Only test for checkpoint/LAS collision if LAS was used. (#3111) Branch: develop https://github.com/wiredtiger/wiredtiger/commit/04cdcbdafd529588d8a4a80ecc308ad01d507a55
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'}

        Message: WT-2953 Only test for checkpoint/LAS collision if LAS was used. (#3111)
        Branch: mongodb-3.4
        https://github.com/wiredtiger/wiredtiger/commit/04cdcbdafd529588d8a4a80ecc308ad01d507a55

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'} Message: WT-2953 Only test for checkpoint/LAS collision if LAS was used. (#3111) Branch: mongodb-3.4 https://github.com/wiredtiger/wiredtiger/commit/04cdcbdafd529588d8a4a80ecc308ad01d507a55
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'}

        Message: WT-2953 Only test for checkpoint/LAS collision if LAS was used. (#3111)
        Branch: mongodb-3.2
        https://github.com/wiredtiger/wiredtiger/commit/04cdcbdafd529588d8a4a80ecc308ad01d507a55

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'} Message: WT-2953 Only test for checkpoint/LAS collision if LAS was used. (#3111) Branch: mongodb-3.2 https://github.com/wiredtiger/wiredtiger/commit/04cdcbdafd529588d8a4a80ecc308ad01d507a55
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

        Message: Import wiredtiger: 74430da40c96247c0a0d92f011ccfb2b175f1f32 from branch mongodb-3.4

        ref: ef9a7983ea..74430da40c
        for: 3.4.0

        SERVER-26753 Minor speed regression (13%) and 'choppy' performance in 3.4 vs 3.2
        WT-2415 Add option for join cursors to return false positives
        WT-2945 Occasional hang running reconfigure fuzz test
        WT-2947 test suite populate functions need refactoring
        WT-2953 Test format failure - stuck cache with 1/42MB dirty
        WT-2961 Add a version drop-down to the web version of the docs
        WT-2964 WTPERF parallel-pop-btree hung in populate phase
        WT-2968 backup cursor opens every file
        WT-2971 Add documentation for raw compression to the WT docs
        WT-2975 Reduce the memory allocated for statistics
        WT-2991 Coverity issue 1365198
        WT-2998 Add error messages to error returns that might be confusing.
        WT-2999 Memory leak in join WT_CURSOR with custom extractor
        WT-3001 WT_EXTENSION_API references are named inconsistently.
        WT-3003 log generation should not auto generate the record and op types
        WT-3005 Add top-level .gitignore file.
        WT-3007 Remove duplicated word in comment
        WT-3008 Run wtperf stress testing against all maintained branches
        WT-3009 Test format hung/aborted with small cache (sub 10mB)
        Branch: master
        https://github.com/mongodb/mongo/commit/0f0db45770f45eab89bd7c2530a259f5ed5b6334

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: Import wiredtiger: 74430da40c96247c0a0d92f011ccfb2b175f1f32 from branch mongodb-3.4 ref: ef9a7983ea..74430da40c for: 3.4.0 SERVER-26753 Minor speed regression (13%) and 'choppy' performance in 3.4 vs 3.2 WT-2415 Add option for join cursors to return false positives WT-2945 Occasional hang running reconfigure fuzz test WT-2947 test suite populate functions need refactoring WT-2953 Test format failure - stuck cache with 1/42MB dirty WT-2961 Add a version drop-down to the web version of the docs WT-2964 WTPERF parallel-pop-btree hung in populate phase WT-2968 backup cursor opens every file WT-2971 Add documentation for raw compression to the WT docs WT-2975 Reduce the memory allocated for statistics WT-2991 Coverity issue 1365198 WT-2998 Add error messages to error returns that might be confusing. WT-2999 Memory leak in join WT_CURSOR with custom extractor WT-3001 WT_EXTENSION_API references are named inconsistently. WT-3003 log generation should not auto generate the record and op types WT-3005 Add top-level .gitignore file. WT-3007 Remove duplicated word in comment WT-3008 Run wtperf stress testing against all maintained branches WT-3009 Test format hung/aborted with small cache (sub 10mB) Branch: master https://github.com/mongodb/mongo/commit/0f0db45770f45eab89bd7c2530a259f5ed5b6334
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'}

        Message: Import wiredtiger: 2e2d5fe23badac038944bdc6639eb62d77136f1a from branch mongodb-3.2

        ref: b11ed312ce..2e2d5fe23b
        for: 3.2.11

        SERVER-26753 Minor speed regression (13%) and 'choppy' performance in 3.4 vs 3.2
        WT-2415 Add option for join cursors to return false positives from a Bloom filter
        WT-2945 Occasional hang running reconfigure fuzz test
        WT-2947 test suite populate functions need refactoring
        WT-2953 Test format failure - stuck cache with 1/42MB dirty
        WT-2961 Add a version drop-down to the web version of the docs
        WT-2964 WTPERF parallel-pop-btree hung in populate phase
        WT-2968 backup cursor opens every file
        WT-2971 Add documentation for raw-compression to the tune_compression page in WT docs
        WT-2975 Reduce the memory allocated for statistics
        WT-2991 Coverity issue 1365198
        WT-2998 Add error messages to error returns that might be confusing.
        WT-2999 Memory leak in join WT_CURSOR with custom extractor
        WT-3001 WT_EXTENSION_API references are named inconsistently.
        WT-3003 log generation should not auto generate the record and op types
        WT-3005 Add top-level .gitignore file.
        WT-3007 Remove duplicated word in comment
        WT-3008 Run wtperf stress testing against all maintained branches
        WT-3009 Test format hung/aborted with small cache (sub 10mB)
        Branch: v3.2
        https://github.com/mongodb/mongo/commit/7c626977cf052314d09648d1061e07c44e964264

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'} Message: Import wiredtiger: 2e2d5fe23badac038944bdc6639eb62d77136f1a from branch mongodb-3.2 ref: b11ed312ce..2e2d5fe23b for: 3.2.11 SERVER-26753 Minor speed regression (13%) and 'choppy' performance in 3.4 vs 3.2 WT-2415 Add option for join cursors to return false positives from a Bloom filter WT-2945 Occasional hang running reconfigure fuzz test WT-2947 test suite populate functions need refactoring WT-2953 Test format failure - stuck cache with 1/42MB dirty WT-2961 Add a version drop-down to the web version of the docs WT-2964 WTPERF parallel-pop-btree hung in populate phase WT-2968 backup cursor opens every file WT-2971 Add documentation for raw-compression to the tune_compression page in WT docs WT-2975 Reduce the memory allocated for statistics WT-2991 Coverity issue 1365198 WT-2998 Add error messages to error returns that might be confusing. WT-2999 Memory leak in join WT_CURSOR with custom extractor WT-3001 WT_EXTENSION_API references are named inconsistently. WT-3003 log generation should not auto generate the record and op types WT-3005 Add top-level .gitignore file. WT-3007 Remove duplicated word in comment WT-3008 Run wtperf stress testing against all maintained branches WT-3009 Test format hung/aborted with small cache (sub 10mB) Branch: v3.2 https://github.com/mongodb/mongo/commit/7c626977cf052314d09648d1061e07c44e964264

          People

          • Assignee:
            david.hows David Hows
            Reporter:
            david.hows David Hows
            Participants:
            Last commenter:
            Ramon Fernandez
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:
              Days since reply:
              24 weeks, 3 days ago
              Date of 1st Reply:

                Agile