Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16670

WiredTiger LSM uses too much CPU when it should be idle

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major - P3
    • Resolution: Unresolved
    • Affects Version/s: 2.8.0-rc3
    • Fix Version/s: Backlog
    • Component/s: Storage, WiredTiger
    • Labels:
    • Operating System:
      ALL

      Description

      This might be the same as SERVER-16356, but 16356 doesn't have enough details and has been closed as fixed without details. The problem is that 12 hours after mongod instance because idle it still uses 3 CPU cores (3 cores are each 40% busy). The server isn't doing much IO per iostat:

      # iostat -kx 10 | grep fioa
      Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
      fioa              0.00     0.00    0.00   12.30     0.00   231.20    37.59     0.00    0.07   0.00   0.00
      fioa              0.00     0.00    0.00    8.10     0.00   128.80    31.80     0.00    0.17   0.00   0.00
      fioa              0.00     0.00    0.00   24.00     0.00   474.80    39.57     0.00    0.78   0.00   0.00
      fioa              0.00     0.00    0.00    4.40     0.00    86.80    39.45     0.00    0.02   0.00   0.00
      fioa              0.00     0.00    0.00    4.00     0.00    57.20    28.60     0.00    0.03   0.00   0.00
      fioa              0.00     0.00    0.00    5.10     0.00    60.80    23.84     0.00    0.00   0.00   0.00
      fioa              0.00     0.00    0.10   12.50     0.40   107.20    17.08     0.00    0.01   0.00   0.00
      fioa              0.00     0.00    0.00    5.20     0.00    62.80    24.15     0.00    0.04   0.00   0.00

      The top-N sources of CPU per "perf record ..."

        79.89%   mongod  libc-2.17.so        [.] __strcmp_sse42                                             
          13.54%   mongod  mongod              [.] __wt_conn_dhandle_close_all                                
           0.85%   mongod  [kernel.kallsyms]   [k] update_sd_lb_stats                                         
           0.25%   mongod  [kernel.kallsyms]   [k] idle_cpu                                                   
           0.25%   mongod  [kernel.kallsyms]   [k] find_next_bit                                              
           0.24%   mongod  [kernel.kallsyms]   [k] _raw_spin_lock                                             
           0.23%   mongod  mongod              [.] __wt_session_get_btree                                     
           0.23%   mongod  mongod              [.] __config_getraw.isra.0                                     
           0.21%   mongod  [kernel.kallsyms]   [k] cpumask_next_and                                           
           0.18%   mongod  libpthread-2.17.so  [.] __lll_lock_wait           

      A thread stack sample shows the busy threads:

      2 threads here -->
      __lll_lock_wait,_L_lock_943,pthread_mutex_lock,__wt_spin_lock,__lsm_drop_file,__wt_lsm_free_chunks,__lsm_worker_general_op,__lsm_worker,start_thread,clone
       
      1 thread here -->
      __strcmp_sse42,__wt_conn_dhandle_close_all,__drop_file,__wt_schema_drop,__lsm_drop_file,__wt_lsm_free_chunks,__lsm_worker_general_op,__lsm_worker,start_thread,clone

        Attachments

          Activity

            People

            • Votes:
              1 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated: