Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-2642

WT hangs when cache is full

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: WT2.8.0
    • Component/s: None
    • None

      Hi!

      We found that sometimes our application hangs (either forever or for a very long time) when inserting data to WT DB.
      Looks like the problem is in insufficient cache size since increasing it helps.
      Moreover I can reliable reproduce the hang by setting small cache size (e.g. 16M when DB is ~ 31M).

      Thread that does the insert has the following stack:

      pthread_cond_timedwait 
      __wt_cond_wait_signal 
      __wt_cond_wait 
      __wt_cache_eviction_worker 
      __wt_cache_eviction_check 
      __cursor_enter 
      __curfile_enter 
      __cursor_func_init 
      __wt_btcur_insert 
      __curfile_insert 
      __curtable_insert
      <application frames>
      

      After that application is unable to open cursors on the table:

       00007f912a7eff4d  __lll_lock_wait + 0x1d
       00007f912a7ebd02  _L_lock_791 + 0xf
       00007f912a7ebc08  pthread_mutex_lock + 0x38
       00007f912baf3f69  __wt_spin_lock + 0x19
       00007f912baf33ec  __schema_add_table + 0x17c
       00007f912baf308b  __wt_schema_get_table + 0x7b
       00007f912ba94b0d  __wt_curtable_open + 0x12d
       00007f912bb01902  __session_open_cursor_int + 0xc2
       00007f912bb04147  __session_open_cursor + 0x437
      

      or

       00007f912a7eda82  pthread_cond_timedwait + 0x132
       00007f912baca42b  __wt_cond_wait_signal + 0x1db
       00007f912ba9ae79  __wt_cond_wait + 0x29
       00007f912ba9a9c5  __wt_cache_eviction_worker + 0x2d5
       00007f912ba08675  __wt_cache_eviction_check + 0x125
       00007f912ba08871  __cursor_enter + 0x31
       00007f912ba084fc  __curfile_enter + 0x3c
       00007f912ba04f01  __cursor_func_init + 0xb1
       00007f912ba04761  __wt_btcur_search + 0x211
       00007f912ba706ea  __curfile_search + 0x31a
       00007f912baf5bd9  __schema_open_table + 0x179
       00007f912baf59f5  __wt_schema_open_table + 0x135
       00007f912baf3423  __schema_add_table + 0x1b3
       00007f912baf308b  __wt_schema_get_table + 0x7b
       00007f912ba94b0d  __wt_curtable_open + 0x12d
       00007f912bb01902  __session_open_cursor_int + 0xc2
       00007f912bb04147  __session_open_cursor + 0x437
      

      I can see this behavior with WT2.8.0, WT2.6.1 and in current 'develop' branch.

      Thanks!

        1. WiredTigerStat.20160520.gz
          109 kB
        2. WiredTigerStat.20160523-json.gz
          108 kB
        3. WiredTigerStat.20160523.gz
          10.61 MB
        4. wt-verbose.log.gz
          536 kB
        5. WiredTigerStat.20160526.gz
          128 kB
        6. cache_dump
          1 kB
        7. stacks
          10 kB
        8. wt-verbose.log.gz
          513 kB
        9. ts2642.jpg
          ts2642.jpg
          200 kB
        10. WiredTigerStat.20160527.gz
          68 kB
        11. cache_dump.256k
          1 kB
        12. stacks.256k
          10 kB
        13. wt-verbose256k.log.gz
          581 kB
        14. test_schema08.py
          4 kB

            Assignee:
            sue.loverso@mongodb.com Susan LoVerso
            Reporter:
            Dmitri Shubin Dmitri Shubin
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: