Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-2945

Occasional hang running reconfigure fuzz test

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: WT2.9.0, 3.2.11, 3.4.0-rc3
    • Labels:
      None
    • Sprint:
      Storage 2016-10-31

      Description

      There have been sporadic test failures from Jenkins since the reconfigure fuzz tester had a timeout added. An example of a failure:

       
      FAIL: test_wt2719_reconfig
      ==========================
       
      configuration timed out: ,shared_cache=(reserve=5MB),eviction=(threads_max=12,threads_min=10),statistics=("all"),shared_cache=(reserve=30MB),eviction=(threads_max=18,threads_min=16),eviction_target=22,shared_cache=(reserve=0),file_manager=(close_handle_minimum=200),file_manager=(close_handle_minimum=200),shared_cache=(chunk=20MB),statistics_log=(timestamp="%H:%M"),async=(threads=7),statistics_log=(wait=60),shared_cache=(chunk=30MB),shared_cache=(name="shared"),checkpoint=(log_size=2),async=(threads=17),log=(prealloc=1),async=(ops_max=2048),statistics_log=(json=0),eviction_target=30,eviction_dirty_trigger=7,checkpoint=(wait=2),shared_cache=(quota=30MB),checkpoint=(wait=100),shared_cache=(quota=30MB),eviction_target=84,eviction=(threads_min=17,threads_max=18),eviction=(threads_min=7,threads_max=10),async=(threads=17),shared_cache=(size=1GB),eviction_dirty_trigger=7
      FAIL test_wt2719_reconfig (exit status: 134)
      

      We should figure out the root cause.

      1. wt2945_stacks.txt
        32 kB
        Alexander Gorrod

        Issue Links

          Activity

          Hide
          alexander.gorrod Alexander Gorrod added a comment -

          The commit here was experimental - we don't have enough evidence to know that it will resolve the issue. I'm going to close the ticket, we can re-open or create a new one if the failure occurs again.

          Show
          alexander.gorrod Alexander Gorrod added a comment - The commit here was experimental - we don't have enough evidence to know that it will resolve the issue. I'm going to close the ticket, we can re-open or create a new one if the failure occurs again.
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}

          Message: WT-2945 Occasional hang running reconfigure fuzz test (#3098)

          • Add a write barrier in front of __wt_cond_signal() to ensure the caller's flags meant to cause a thread to exit are seen by the thread.
          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'} Message: WT-2945 Occasional hang running reconfigure fuzz test (#3098) Add a write barrier in front of __wt_cond_signal() to ensure the caller's flags meant to cause a thread to exit are seen by the thread. Make the LSM start/stop worker thread loops look the same. Branch: mongodb-3.4 https://github.com/wiredtiger/wiredtiger/commit/b6be49dd39979e55bf809111c71dce0389f6b728
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}

          Message: WT-2945 Occasional hang running reconfigure fuzz test (#3098)

          • Add a write barrier in front of __wt_cond_signal() to ensure the caller's flags meant to cause a thread to exit are seen by the thread.
          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'} Message: WT-2945 Occasional hang running reconfigure fuzz test (#3098) Add a write barrier in front of __wt_cond_signal() to ensure the caller's flags meant to cause a thread to exit are seen by the thread. Make the LSM start/stop worker thread loops look the same. Branch: mongodb-3.2 https://github.com/wiredtiger/wiredtiger/commit/b6be49dd39979e55bf809111c71dce0389f6b728
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: Import wiredtiger: 74430da40c96247c0a0d92f011ccfb2b175f1f32 from branch mongodb-3.4

          ref: ef9a7983ea..74430da40c
          for: 3.4.0

          SERVER-26753 Minor speed regression (13%) and 'choppy' performance in 3.4 vs 3.2
          WT-2415 Add option for join cursors to return false positives
          WT-2945 Occasional hang running reconfigure fuzz test
          WT-2947 test suite populate functions need refactoring
          WT-2953 Test format failure - stuck cache with 1/42MB dirty
          WT-2961 Add a version drop-down to the web version of the docs
          WT-2964 WTPERF parallel-pop-btree hung in populate phase
          WT-2968 backup cursor opens every file
          WT-2971 Add documentation for raw compression to the WT docs
          WT-2975 Reduce the memory allocated for statistics
          WT-2991 Coverity issue 1365198
          WT-2998 Add error messages to error returns that might be confusing.
          WT-2999 Memory leak in join WT_CURSOR with custom extractor
          WT-3001 WT_EXTENSION_API references are named inconsistently.
          WT-3003 log generation should not auto generate the record and op types
          WT-3005 Add top-level .gitignore file.
          WT-3007 Remove duplicated word in comment
          WT-3008 Run wtperf stress testing against all maintained branches
          WT-3009 Test format hung/aborted with small cache (sub 10mB)
          Branch: master
          https://github.com/mongodb/mongo/commit/0f0db45770f45eab89bd7c2530a259f5ed5b6334

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: Import wiredtiger: 74430da40c96247c0a0d92f011ccfb2b175f1f32 from branch mongodb-3.4 ref: ef9a7983ea..74430da40c for: 3.4.0 SERVER-26753 Minor speed regression (13%) and 'choppy' performance in 3.4 vs 3.2 WT-2415 Add option for join cursors to return false positives WT-2945 Occasional hang running reconfigure fuzz test WT-2947 test suite populate functions need refactoring WT-2953 Test format failure - stuck cache with 1/42MB dirty WT-2961 Add a version drop-down to the web version of the docs WT-2964 WTPERF parallel-pop-btree hung in populate phase WT-2968 backup cursor opens every file WT-2971 Add documentation for raw compression to the WT docs WT-2975 Reduce the memory allocated for statistics WT-2991 Coverity issue 1365198 WT-2998 Add error messages to error returns that might be confusing. WT-2999 Memory leak in join WT_CURSOR with custom extractor WT-3001 WT_EXTENSION_API references are named inconsistently. WT-3003 log generation should not auto generate the record and op types WT-3005 Add top-level .gitignore file. WT-3007 Remove duplicated word in comment WT-3008 Run wtperf stress testing against all maintained branches WT-3009 Test format hung/aborted with small cache (sub 10mB) Branch: master https://github.com/mongodb/mongo/commit/0f0db45770f45eab89bd7c2530a259f5ed5b6334
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'}

          Message: Import wiredtiger: 2e2d5fe23badac038944bdc6639eb62d77136f1a from branch mongodb-3.2

          ref: b11ed312ce..2e2d5fe23b
          for: 3.2.11

          SERVER-26753 Minor speed regression (13%) and 'choppy' performance in 3.4 vs 3.2
          WT-2415 Add option for join cursors to return false positives from a Bloom filter
          WT-2945 Occasional hang running reconfigure fuzz test
          WT-2947 test suite populate functions need refactoring
          WT-2953 Test format failure - stuck cache with 1/42MB dirty
          WT-2961 Add a version drop-down to the web version of the docs
          WT-2964 WTPERF parallel-pop-btree hung in populate phase
          WT-2968 backup cursor opens every file
          WT-2971 Add documentation for raw-compression to the tune_compression page in WT docs
          WT-2975 Reduce the memory allocated for statistics
          WT-2991 Coverity issue 1365198
          WT-2998 Add error messages to error returns that might be confusing.
          WT-2999 Memory leak in join WT_CURSOR with custom extractor
          WT-3001 WT_EXTENSION_API references are named inconsistently.
          WT-3003 log generation should not auto generate the record and op types
          WT-3005 Add top-level .gitignore file.
          WT-3007 Remove duplicated word in comment
          WT-3008 Run wtperf stress testing against all maintained branches
          WT-3009 Test format hung/aborted with small cache (sub 10mB)
          Branch: v3.2
          https://github.com/mongodb/mongo/commit/7c626977cf052314d09648d1061e07c44e964264

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'} Message: Import wiredtiger: 2e2d5fe23badac038944bdc6639eb62d77136f1a from branch mongodb-3.2 ref: b11ed312ce..2e2d5fe23b for: 3.2.11 SERVER-26753 Minor speed regression (13%) and 'choppy' performance in 3.4 vs 3.2 WT-2415 Add option for join cursors to return false positives from a Bloom filter WT-2945 Occasional hang running reconfigure fuzz test WT-2947 test suite populate functions need refactoring WT-2953 Test format failure - stuck cache with 1/42MB dirty WT-2961 Add a version drop-down to the web version of the docs WT-2964 WTPERF parallel-pop-btree hung in populate phase WT-2968 backup cursor opens every file WT-2971 Add documentation for raw-compression to the tune_compression page in WT docs WT-2975 Reduce the memory allocated for statistics WT-2991 Coverity issue 1365198 WT-2998 Add error messages to error returns that might be confusing. WT-2999 Memory leak in join WT_CURSOR with custom extractor WT-3001 WT_EXTENSION_API references are named inconsistently. WT-3003 log generation should not auto generate the record and op types WT-3005 Add top-level .gitignore file. WT-3007 Remove duplicated word in comment WT-3008 Run wtperf stress testing against all maintained branches WT-3009 Test format hung/aborted with small cache (sub 10mB) Branch: v3.2 https://github.com/mongodb/mongo/commit/7c626977cf052314d09648d1061e07c44e964264

            People

            • Assignee:
              keith.bostic Keith Bostic
              Reporter:
              alexander.gorrod Alexander Gorrod
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                  Agile