Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-2642

WT hangs when cache is full

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Done
    • Affects Version/s: WT2.8.0
    • Fix Version/s: None
    • Labels:
      None

      Description

      Hi!

      We found that sometimes our application hangs (either forever or for a very long time) when inserting data to WT DB.
      Looks like the problem is in insufficient cache size since increasing it helps.
      Moreover I can reliable reproduce the hang by setting small cache size (e.g. 16M when DB is ~ 31M).

      Thread that does the insert has the following stack:

      pthread_cond_timedwait 
      __wt_cond_wait_signal 
      __wt_cond_wait 
      __wt_cache_eviction_worker 
      __wt_cache_eviction_check 
      __cursor_enter 
      __curfile_enter 
      __cursor_func_init 
      __wt_btcur_insert 
      __curfile_insert 
      __curtable_insert
      		 
      <application frames>

      After that application is unable to open cursors on the table:

       00007f912a7eff4d  __lll_lock_wait + 0x1d
      		 
       00007f912a7ebd02  _L_lock_791 + 0xf
      		 
       00007f912a7ebc08  pthread_mutex_lock + 0x38
      		 
       00007f912baf3f69  __wt_spin_lock + 0x19
      		 
       00007f912baf33ec  __schema_add_table + 0x17c
      		 
       00007f912baf308b  __wt_schema_get_table + 0x7b
      		 
       00007f912ba94b0d  __wt_curtable_open + 0x12d
      		 
       00007f912bb01902  __session_open_cursor_int + 0xc2
      		 
       00007f912bb04147  __session_open_cursor + 0x437

      or

       00007f912a7eda82  pthread_cond_timedwait + 0x132
      		 
       00007f912baca42b  __wt_cond_wait_signal + 0x1db
      		 
       00007f912ba9ae79  __wt_cond_wait + 0x29
      		 
       00007f912ba9a9c5  __wt_cache_eviction_worker + 0x2d5
      		 
       00007f912ba08675  __wt_cache_eviction_check + 0x125
      		 
       00007f912ba08871  __cursor_enter + 0x31
      		 
       00007f912ba084fc  __curfile_enter + 0x3c
      		 
       00007f912ba04f01  __cursor_func_init + 0xb1
      		 
       00007f912ba04761  __wt_btcur_search + 0x211
      		 
       00007f912ba706ea  __curfile_search + 0x31a
      		 
       00007f912baf5bd9  __schema_open_table + 0x179
      		 
       00007f912baf59f5  __wt_schema_open_table + 0x135
      		 
       00007f912baf3423  __schema_add_table + 0x1b3
      		 
       00007f912baf308b  __wt_schema_get_table + 0x7b
      		 
       00007f912ba94b0d  __wt_curtable_open + 0x12d
      		 
       00007f912bb01902  __session_open_cursor_int + 0xc2
      		 
       00007f912bb04147  __session_open_cursor + 0x437

      I can see this behavior with WT2.8.0, WT2.6.1 and in current 'develop' branch.

      Thanks!

        Attachments

        1. cache_dump
          1 kB
        2. cache_dump.256k
          1 kB
        3. stacks
          10 kB
        4. stacks.256k
          10 kB
        5. test_schema08.py
          4 kB
        6. ts2642.jpg
          ts2642.jpg
          200 kB
        7. WiredTigerStat.20160520.gz
          109 kB
        8. WiredTigerStat.20160523.gz
          10.61 MB
        9. WiredTigerStat.20160523-json.gz
          108 kB
        10. WiredTigerStat.20160526.gz
          128 kB
        11. WiredTigerStat.20160527.gz
          68 kB
        12. wt-verbose.log.gz
          513 kB
        13. wt-verbose.log.gz
          536 kB
        14. wt-verbose256k.log.gz
          581 kB

          Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. WT-3214 workload stuck due to eviction of dirty data

          • Major - P3 - Major loss of function.
          • Closed
          related to

          Bug - A problem which impairs or prevents the functions of the product. WT-2660 Hang between eviction and connection close

          • Major - P3 - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. WT-2678 The metadata should not imply that an empty value is true

          • Major - P3 - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. WT-2452 Implement a mechanism that gives priority in cache to some tables

          • Major - P3 - Major loss of function.
          • Closed

            Activity

              People

              • Assignee:
                sue.loverso Sue LoVerso
                Reporter:
                Dmitri Shubin Dmitri Shubin
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: