Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-3024

wtperf medium-lsm-compact test can hang

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: WT2.9.0, 3.4.0-rc4, 3.2.12
    • Labels:
      None
    • # Replies:
      5
    • Last comment by Customer:
      true
    • Sprint:
      Storage 2016-11-21

      Description

      Repro with

      ./bench/wtperf/wtperf -O ../bench/wtperf/runners/medium-lsm-compact.wtperf -o verbose=2
      

      Output:

      Starting 1 populate thread(s) for 50000000 items
      7831255 populate inserts (7831255 of 50000000) in 5 secs (5 total secs)
      7864217 populate inserts (15695472 of 50000000) in 5 secs (10 total secs)
      7865863 populate inserts (23561335 of 50000000) in 5 secs (15 total secs)
      7914062 populate inserts (31475397 of 50000000) in 5 secs (20 total secs)
      7838969 populate inserts (39314366 of 50000000) in 5 secs (25 total secs)
      7811496 populate inserts (47125862 of 50000000) in 5 secs (30 total secs)
      Finished load of 50000000 items
      Load time: 32.05
      load ops/sec: 1560305
      Compact after populate
      Compact completed in 0 seconds
      

      From GDB it appears that this is stuck in the close_reopen call in WTPERF, doing a __conn_close

      Thread 1 (Thread 0x7f3559a82780 (LWP 32750)):
      #0  0x00007f35589f1b27 in sched_yield () from /lib64/libc.so.6
      #1  0x000000000043e3f5 in __wt_yield () at ../src/os_posix/os_yield.c:18
      #2  0x0000000000418150 in __wt_connection_close (conn=conn@entry=0x15c4000) at ../src/conn/conn_open.c:101
      #3  0x000000000040efa6 in __conn_close (wt_conn=0x15c4000, config=0x0) at ../src/conn/conn_api.c:1089
      #4  0x0000000000409051 in close_reopen (wtperf=0x7fff97ac4600) at ../../../bench/wtperf/wtperf.c:1605
      #5  start_run (wtperf=wtperf@entry=0x7fff97ac4600) at ../../../bench/wtperf/wtperf.c:2229
      #6  0x00000000004051c1 in start_all_runs (wtperf=0x7fff97ac4600) at ../../../bench/wtperf/wtperf.c:2109
      #7  main (argc=<optimized out>, argv=<optimized out>) at ../../../bench/wtperf/wtperf.c:2598
      

      The issue appears to be that txn_global->metadata_pinned (7 in my test) is below txn_global->current (9 in test) and txn_global->oldest_id (9 in test).

        Issue Links

          Activity

          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: WT-3024 Fix a hang on close caused by leaving a transaction ID pinned. (#3152)
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/1ce36af292fd0317021091a0523376b0533ac3bf

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: WT-3024 Fix a hang on close caused by leaving a transaction ID pinned. (#3152) Branch: develop https://github.com/wiredtiger/wiredtiger/commit/1ce36af292fd0317021091a0523376b0533ac3bf
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: WT-3024 Fix a hang on close caused by leaving a transaction ID pinned. (#3152)
          Branch: mongodb-3.4
          https://github.com/wiredtiger/wiredtiger/commit/1ce36af292fd0317021091a0523376b0533ac3bf

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: WT-3024 Fix a hang on close caused by leaving a transaction ID pinned. (#3152) Branch: mongodb-3.4 https://github.com/wiredtiger/wiredtiger/commit/1ce36af292fd0317021091a0523376b0533ac3bf
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: Import wiredtiger: ca6eee06ffdacc8e191987e64b3791740dad21e1 from branch mongodb-3.4

          ref: 74430da40c..ca6eee06ff
          for: 3.4.0

          WT-2962 Provide a way to configure builtin extensions
          WT-2984 Search of metadata for recently created collection gets WT_NOTFOUND
          WT-3000 Missing log records in recovery when crashing after a log file switch
          WT-3002 Allow applications to exempt threads from eviction.
          WT-3004 lint: declare functions that don't return a value as void
          WT-3011 __wt_curjoin_open() saves the wrong URI in the cursor.
          WT-3012 Test format hanging on LSM configurations
          WT-3015 Test format stuck with 2mb cache
          WT-3016 Tests needed for systems without ftruncate
          WT-3017 Hazard pointer race with page replace causes error
          WT-3018 lint
          WT-3020 LSM primary changes impact parallel-pop-lsm load time
          WT-3022 LSM operations get stuck in __wt_clsm_await_switch waiting for switch on tree to complete
          WT-3023 Test format hang on zSeries
          WT-3024 wtperf medium-lsm-compact test can hang
          Branch: master
          https://github.com/mongodb/mongo/commit/fb4ae3792065e98696e391ac1c4602216b8502cb

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: Import wiredtiger: ca6eee06ffdacc8e191987e64b3791740dad21e1 from branch mongodb-3.4 ref: 74430da40c..ca6eee06ff for: 3.4.0 WT-2962 Provide a way to configure builtin extensions WT-2984 Search of metadata for recently created collection gets WT_NOTFOUND WT-3000 Missing log records in recovery when crashing after a log file switch WT-3002 Allow applications to exempt threads from eviction. WT-3004 lint: declare functions that don't return a value as void WT-3011 __wt_curjoin_open() saves the wrong URI in the cursor. WT-3012 Test format hanging on LSM configurations WT-3015 Test format stuck with 2mb cache WT-3016 Tests needed for systems without ftruncate WT-3017 Hazard pointer race with page replace causes error WT-3018 lint WT-3020 LSM primary changes impact parallel-pop-lsm load time WT-3022 LSM operations get stuck in __wt_clsm_await_switch waiting for switch on tree to complete WT-3023 Test format hang on zSeries WT-3024 wtperf medium-lsm-compact test can hang Branch: master https://github.com/mongodb/mongo/commit/fb4ae3792065e98696e391ac1c4602216b8502cb
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: WT-3024 Fix a hang on close caused by leaving a transaction ID pinned. (#3152)
          Branch: mongodb-3.2
          https://github.com/wiredtiger/wiredtiger/commit/1ce36af292fd0317021091a0523376b0533ac3bf

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: WT-3024 Fix a hang on close caused by leaving a transaction ID pinned. (#3152) Branch: mongodb-3.2 https://github.com/wiredtiger/wiredtiger/commit/1ce36af292fd0317021091a0523376b0533ac3bf
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: Import wiredtiger: 040e3d6f764c0fb626cb47fede54469f57d0c6e0 from branch mongodb-3.2

          ref: 187707a5c1..040e3d6f76
          for: 3.2.12

          WT-2962 Provide a way to configure builtin extensions
          WT-2984 Search of metadata for recently created collection gets WT_NOTFOUND
          WT-3000 Missing log records in recovery when crashing after a log file switch
          WT-3002 Allow applications to exempt threads from eviction.
          WT-3004 lint: declare functions that don't return a value as void
          WT-3011 __wt_curjoin_open() saves the wrong URI in the cursor.
          WT-3012 Test format hanging on LSM configurations
          WT-3015 Test format stuck with 2mb cache
          WT-3016 Tests needed for systems without ftruncate
          WT-3017 Hazard pointer race with page replace causes error
          WT-3018 lint
          WT-3020 LSM primary changes impact parallel-pop-lsm load time
          WT-3022 LSM operations get stuck in __wt_clsm_await_switch waiting for switch on tree to complete
          WT-3023 Test format hang on zSeries
          WT-3024 wtperf medium-lsm-compact test can hang
          Branch: v3.2
          https://github.com/mongodb/mongo/commit/c586934f7212f6a9a2087cbaf9a8fcd7d7ce9abf

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: Import wiredtiger: 040e3d6f764c0fb626cb47fede54469f57d0c6e0 from branch mongodb-3.2 ref: 187707a5c1..040e3d6f76 for: 3.2.12 WT-2962 Provide a way to configure builtin extensions WT-2984 Search of metadata for recently created collection gets WT_NOTFOUND WT-3000 Missing log records in recovery when crashing after a log file switch WT-3002 Allow applications to exempt threads from eviction. WT-3004 lint: declare functions that don't return a value as void WT-3011 __wt_curjoin_open() saves the wrong URI in the cursor. WT-3012 Test format hanging on LSM configurations WT-3015 Test format stuck with 2mb cache WT-3016 Tests needed for systems without ftruncate WT-3017 Hazard pointer race with page replace causes error WT-3018 lint WT-3020 LSM primary changes impact parallel-pop-lsm load time WT-3022 LSM operations get stuck in __wt_clsm_await_switch waiting for switch on tree to complete WT-3023 Test format hang on zSeries WT-3024 wtperf medium-lsm-compact test can hang Branch: v3.2 https://github.com/mongodb/mongo/commit/c586934f7212f6a9a2087cbaf9a8fcd7d7ce9abf

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since reply:
                27 weeks, 6 days ago
                Date of 1st Reply:

                  Agile