Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-3078

Fix a hang in the reconfiguration test.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.2.12, WT2.9.1, 3.5.2, 3.4.2
    • Labels:
      None
    • # Replies:
      6
    • Last comment by Customer:
      true

      Description

      Reconfiguration test failure on tinderbox (link here):

      % cd /home/jenkins/jenkins/workspace/wiredtiger-pull-request-linux/build_posix/test/csuite
      % gdb test_wt2719_reconfig core.test_wt2719_rec.29270
       
      Thread 13 (Thread 0x7f5e022fd700 (LWP 30127)):
      #0  0x0000003e6ce0c8e9 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
         from /lib64/libpthread.so.0
      #1  0x00007f5e0d49c2ca in __wt_cond_wait_signal (session=0x16f22c0,
          cond=0x1716140, usecs=60000000, signalled=0x7f5e022fceaf)
          at ../src/os_posix/os_mtx_cond.c:71
      #2  0x00007f5e0d430f6e in __wt_cond_wait (session=0x16f22c0, cond=0x1716140,
          usecs=60000000) at ../src/include/misc.i:18
      #3  0x00007f5e0d4327f4 in __statlog_server (arg=0x16f22c0)
          at ../src/conn/conn_stat.c:531
      #4  0x0000003e6ce07555 in start_thread () from /lib64/libpthread.so.0
      #5  0x0000003e6cb02ded in clone () from /lib64/libc.so.6
       
      Thread 1 (Thread 0x7f5e0d38e700 (LWP 29270)):
      #0  0x0000003e6ca349c8 in raise () from /lib64/libc.so.6
      #1  0x0000003e6ca3665a in abort () from /lib64/libc.so.6
      #2  0x000000000040106d in on_alarm (signo=14)
          at ../../../test/csuite/wt2719_reconfig/main.c:203
      #3  <signal handler called>
      #4  0x0000003e6ce0859b in pthread_join () from /lib64/libpthread.so.0
      #5  0x00007f5e0d49c7ee in __wt_thread_join (session=0x16ebe00,
          tid=140041740343040) at ../src/os_posix/os_thread.c:37
      #6  0x00007f5e0d432ae5 in __wt_statlog_destroy (session=0x16ebe00,
          is_close=false) at ../src/conn/conn_stat.c:639
      #7  0x00007f5e0d432a08 in __wt_statlog_create (session=0x16ebe00,
          cfg=0x7ffd1f9e8000) at ../src/conn/conn_stat.c:613
      #8  0x00007f5e0d423561 in __conn_reconfigure (wt_conn=0x16d60a0,
          config=0x17b9160 ",async=(threads=10),cache_size=214MB,verbose=(\"mutex\"),eviction_dirty_trigger=37,statistics_log=(wait=60),async=(enabled=1),eviction_target=84,checkpoint=(wait=100),statistics_log=(wait=76),verbose=(\""...)
          at ../src/conn/conn_api.c:1143
      #9  0x00000000004010b6 in reconfig (opts=0x7ffd1f9e80d0, session=0x16ed180,
          config=0x17b9160 ",async=(threads=10),cache_size=214MB,verbose=(\"mutex\"),eviction_dirty_trigger=37,statistics_log=(wait=60),async=(enabled=1),eviction_target=84,checkpoint=(wait=100),statistics_log=(wait=76),verbose=(\""...)
          at ../../../test/csuite/wt2719_reconfig/main.c:221
      #10 0x00000000004014be in main (argc=1, argv=0x7ffd1f9e8278)
          at ../../../test/csuite/wt2719_reconfig/main.c:300
      

        Issue Links

          Activity

          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}

          Message: WT-3078 test reconfiguration hang in the statlog server. (#3198)

          Flush the caller's flags before reading the condition information, we may not write anything before joining the exiting thread, and a write-barrier() would not necessarily have any effect.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/a026912444c9714011acfec0ca40722e3b55e2a1

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'} Message: WT-3078 test reconfiguration hang in the statlog server. (#3198) Flush the caller's flags before reading the condition information, we may not write anything before joining the exiting thread, and a write-barrier() would not necessarily have any effect. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/a026912444c9714011acfec0ca40722e3b55e2a1
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: Import wiredtiger: 8d2324943364286056ae399043f70b8a937de312 from branch mongodb-3.6

          ref: 1b6c815a3f..8d23249433
          for: 3.5.2

          SERVER-26545 Remove fixed-size limitation on WiredTiger hazard pointers
          WT-2402 Misaligned structure accesses lead to undefined behavior
          WT-283 Add a way to change persistent object settings
          WT-3056 For cursors with projections, keys should be allowed
          WT-3061 syscall test runs with checkpoint_sync=false and doesn't acknowledge pwrite64
          WT-3066 lint
          WT-3068 Copy wtperf artifacts when running Jenkins tests
          WT-3069 Fix build failures in LevelDB APIs
          WT-3070 Fix search_near() for index cursor
          WT-3071 Java: fix build with -Werror=sign-conversion
          WT-3075 Document and enforce that WiredTiger now depends on Python 2.7
          WT-3078 Fix a hang in the reconfiguration test.
          WT-3084 Fix Coverity resource leak complaint.
          Branch: master
          https://github.com/mongodb/mongo/commit/4a185996c83862c2e4b911ce637591c4ff535108

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: Import wiredtiger: 8d2324943364286056ae399043f70b8a937de312 from branch mongodb-3.6 ref: 1b6c815a3f..8d23249433 for: 3.5.2 SERVER-26545 Remove fixed-size limitation on WiredTiger hazard pointers WT-2402 Misaligned structure accesses lead to undefined behavior WT-283 Add a way to change persistent object settings WT-3056 For cursors with projections, keys should be allowed WT-3061 syscall test runs with checkpoint_sync=false and doesn't acknowledge pwrite64 WT-3066 lint WT-3068 Copy wtperf artifacts when running Jenkins tests WT-3069 Fix build failures in LevelDB APIs WT-3070 Fix search_near() for index cursor WT-3071 Java: fix build with -Werror=sign-conversion WT-3075 Document and enforce that WiredTiger now depends on Python 2.7 WT-3078 Fix a hang in the reconfiguration test. WT-3084 Fix Coverity resource leak complaint. Branch: master https://github.com/mongodb/mongo/commit/4a185996c83862c2e4b911ce637591c4ff535108
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}

          Message: WT-3078 test reconfiguration hang in the statlog server. (#3198)

          Flush the caller's flags before reading the condition information, we may not write anything before joining the exiting thread, and a write-barrier() would not necessarily have any effect.
          Branch: mongodb-3.4
          https://github.com/wiredtiger/wiredtiger/commit/a026912444c9714011acfec0ca40722e3b55e2a1

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'} Message: WT-3078 test reconfiguration hang in the statlog server. (#3198) Flush the caller's flags before reading the condition information, we may not write anything before joining the exiting thread, and a write-barrier() would not necessarily have any effect. Branch: mongodb-3.4 https://github.com/wiredtiger/wiredtiger/commit/a026912444c9714011acfec0ca40722e3b55e2a1
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}

          Message: WT-3078 test reconfiguration hang in the statlog server. (#3198)

          Flush the caller's flags before reading the condition information, we may not write anything before joining the exiting thread, and a write-barrier() would not necessarily have any effect.
          Branch: mongodb-3.2
          https://github.com/wiredtiger/wiredtiger/commit/a026912444c9714011acfec0ca40722e3b55e2a1

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'} Message: WT-3078 test reconfiguration hang in the statlog server. (#3198) Flush the caller's flags before reading the condition information, we may not write anything before joining the exiting thread, and a write-barrier() would not necessarily have any effect. Branch: mongodb-3.2 https://github.com/wiredtiger/wiredtiger/commit/a026912444c9714011acfec0ca40722e3b55e2a1
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'}

          Message: Import wiredtiger: d48181f6f4db08761ed7b80b0332908b272ad0d0 from branch mongodb-3.2

          ref: 040e3d6f76..d48181f6f4
          for: 3.2.12

          SERVER-26545 Remove fixed-size limitation on WiredTiger hazard pointers
          WT-2336 Add a test validating schema operations via file system call monitoring
          WT-2402 Misaligned structure accesses lead to undefined behavior
          WT-2670 Inefficient I/O when read full DB (poor readahead)
          WT-283 Add a way to change persistent object settings
          WT-2960 Inserting multi-megabyte values can cause pathological lookaside usage
          WT-2969 Possible snapshot corruption during compaction
          WT-3014 Add GCC/clang support for ELF symbol visibility.
          WT-3021 Fixes needed for Java log cursor example, Java raw mode cursors, log cursors in raw mode
          WT-3025 fix error path in log_force_sync
          WT-3028 Workloads with all dirty pages could trigger diagnostic stuck check
          WT-3030 Test failure indicating invalid key order during traversal
          WT-3034 Add support for single-writer named snapshots.
          WT-3037 Fix some outdated comments in logging
          WT-3048 WiredTiger maximum size warning uses the wrong format.
          WT-3051 Remove external __wt_hex symbol.
          WT-3052 Improve search if an index hint is wrong
          WT-3053 Review Python and Java calls to internal WiredTiger functions
          WT-3054 Java PackTest, PackTest03 do not compile
          WT-3055 Java AsyncTest faults
          WT-3056 For cursors with projections, keys should be allowed
          WT-3057 WiredTiger hazard pointers should use the WT_REF, not the WT_PAGE.
          WT-3061 syscall test runs with checkpoint_sync=false and doesn't acknowledge pwrite64
          WT-3064 minor tree cleanups: .gitignore, NEWS misspelling
          WT-3066 lint
          WT-3068 Copy wtperf artifacts when running Jenkins tests
          WT-3069 Fix build failures in LevelDB APIs
          WT-3070 Fix search_near() for index cursor
          WT-3071 Java: fix build with -Werror=sign-conversion
          WT-3075 Document and enforce that WiredTiger now depends on Python 2.7
          WT-3078 Fix a hang in the reconfiguration test.
          WT-3084 Fix Coverity resource leak complaint.
          Branch: v3.2
          https://github.com/mongodb/mongo/commit/52b68fa86ea43e909ad42c901d0579bced6b205f

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'} Message: Import wiredtiger: d48181f6f4db08761ed7b80b0332908b272ad0d0 from branch mongodb-3.2 ref: 040e3d6f76..d48181f6f4 for: 3.2.12 SERVER-26545 Remove fixed-size limitation on WiredTiger hazard pointers WT-2336 Add a test validating schema operations via file system call monitoring WT-2402 Misaligned structure accesses lead to undefined behavior WT-2670 Inefficient I/O when read full DB (poor readahead) WT-283 Add a way to change persistent object settings WT-2960 Inserting multi-megabyte values can cause pathological lookaside usage WT-2969 Possible snapshot corruption during compaction WT-3014 Add GCC/clang support for ELF symbol visibility. WT-3021 Fixes needed for Java log cursor example, Java raw mode cursors, log cursors in raw mode WT-3025 fix error path in log_force_sync WT-3028 Workloads with all dirty pages could trigger diagnostic stuck check WT-3030 Test failure indicating invalid key order during traversal WT-3034 Add support for single-writer named snapshots. WT-3037 Fix some outdated comments in logging WT-3048 WiredTiger maximum size warning uses the wrong format. WT-3051 Remove external __wt_hex symbol. WT-3052 Improve search if an index hint is wrong WT-3053 Review Python and Java calls to internal WiredTiger functions WT-3054 Java PackTest, PackTest03 do not compile WT-3055 Java AsyncTest faults WT-3056 For cursors with projections, keys should be allowed WT-3057 WiredTiger hazard pointers should use the WT_REF, not the WT_PAGE. WT-3061 syscall test runs with checkpoint_sync=false and doesn't acknowledge pwrite64 WT-3064 minor tree cleanups: .gitignore, NEWS misspelling WT-3066 lint WT-3068 Copy wtperf artifacts when running Jenkins tests WT-3069 Fix build failures in LevelDB APIs WT-3070 Fix search_near() for index cursor WT-3071 Java: fix build with -Werror=sign-conversion WT-3075 Document and enforce that WiredTiger now depends on Python 2.7 WT-3078 Fix a hang in the reconfiguration test. WT-3084 Fix Coverity resource leak complaint. Branch: v3.2 https://github.com/mongodb/mongo/commit/52b68fa86ea43e909ad42c901d0579bced6b205f
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'}

          Message: Import wiredtiger: 8d2324943364286056ae399043f70b8a937de312 from branch mongodb-3.4

          ref: ca6eee06ff..8d23249433
          for: 3.4.2

          SERVER-26545 Remove fixed-size limitation on WiredTiger hazard pointers
          WT-2336 Add a test validating schema operations via file system call monitoring
          WT-2402 Misaligned structure accesses lead to undefined behavior
          WT-2670 Inefficient I/O when read full DB (poor readahead)
          WT-283 Add a way to change persistent object settings
          WT-2960 Inserting multi-megabyte values can cause pathological lookaside usage
          WT-2969 Possible snapshot corruption during compaction
          WT-3014 Add GCC/clang support for ELF symbol visibility.
          WT-3021 Fixes needed for Java log cursor example, Java raw mode cursors, log cursors in raw mode
          WT-3025 fix error path in log_force_sync
          WT-3028 Workloads with all dirty pages could trigger diagnostic stuck check
          WT-3030 Test failure indicating invalid key order during traversal
          WT-3034 Add support for single-writer named snapshots.
          WT-3037 Fix some outdated comments in logging
          WT-3048 WiredTiger maximum size warning uses the wrong format.
          WT-3051 Remove external __wt_hex symbol.
          WT-3052 Improve search if an index hint is wrong
          WT-3053 Review Python and Java calls to internal WiredTiger functions
          WT-3054 Java PackTest, PackTest03 do not compile
          WT-3055 Java AsyncTest faults
          WT-3056 For cursors with projections, keys should be allowed
          WT-3057 WiredTiger hazard pointers should use the WT_REF, not the WT_PAGE.
          WT-3061 syscall test runs with checkpoint_sync=false and doesn't acknowledge pwrite64
          WT-3064 minor tree cleanups: .gitignore, NEWS misspelling
          WT-3066 lint
          WT-3068 Copy wtperf artifacts when running Jenkins tests
          WT-3069 Fix build failures in LevelDB APIs
          WT-3070 Fix search_near() for index cursor
          WT-3071 Java: fix build with -Werror=sign-conversion
          WT-3075 Document and enforce that WiredTiger now depends on Python 2.7
          WT-3078 Fix a hang in the reconfiguration test.
          WT-3084 Fix Coverity resource leak complaint.
          Branch: v3.4
          https://github.com/mongodb/mongo/commit/d2c64ac8c526b70eadeb859ec41370a5f03a64aa

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'} Message: Import wiredtiger: 8d2324943364286056ae399043f70b8a937de312 from branch mongodb-3.4 ref: ca6eee06ff..8d23249433 for: 3.4.2 SERVER-26545 Remove fixed-size limitation on WiredTiger hazard pointers WT-2336 Add a test validating schema operations via file system call monitoring WT-2402 Misaligned structure accesses lead to undefined behavior WT-2670 Inefficient I/O when read full DB (poor readahead) WT-283 Add a way to change persistent object settings WT-2960 Inserting multi-megabyte values can cause pathological lookaside usage WT-2969 Possible snapshot corruption during compaction WT-3014 Add GCC/clang support for ELF symbol visibility. WT-3021 Fixes needed for Java log cursor example, Java raw mode cursors, log cursors in raw mode WT-3025 fix error path in log_force_sync WT-3028 Workloads with all dirty pages could trigger diagnostic stuck check WT-3030 Test failure indicating invalid key order during traversal WT-3034 Add support for single-writer named snapshots. WT-3037 Fix some outdated comments in logging WT-3048 WiredTiger maximum size warning uses the wrong format. WT-3051 Remove external __wt_hex symbol. WT-3052 Improve search if an index hint is wrong WT-3053 Review Python and Java calls to internal WiredTiger functions WT-3054 Java PackTest, PackTest03 do not compile WT-3055 Java AsyncTest faults WT-3056 For cursors with projections, keys should be allowed WT-3057 WiredTiger hazard pointers should use the WT_REF, not the WT_PAGE. WT-3061 syscall test runs with checkpoint_sync=false and doesn't acknowledge pwrite64 WT-3064 minor tree cleanups: .gitignore, NEWS misspelling WT-3066 lint WT-3068 Copy wtperf artifacts when running Jenkins tests WT-3069 Fix build failures in LevelDB APIs WT-3070 Fix search_near() for index cursor WT-3071 Java: fix build with -Werror=sign-conversion WT-3075 Document and enforce that WiredTiger now depends on Python 2.7 WT-3078 Fix a hang in the reconfiguration test. WT-3084 Fix Coverity resource leak complaint. Branch: v3.4 https://github.com/mongodb/mongo/commit/d2c64ac8c526b70eadeb859ec41370a5f03a64aa

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since reply:
                19 weeks, 4 days ago
                Date of 1st Reply: