Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-31704

Periodic drops in throughput during checkpoints while waiting on schema lock

    XMLWordPrintableJSON

Details

    • ALL
    • Hide

      Run mix.js
      Set the following parameter:

      wiredTigerEngineRuntimeConfig:"file_manager=(close_handle_minimum=100,close_idle_time=300,close_scan_interval=30)"

      Show
      Run mix.js Set the following parameter: wiredTigerEngineRuntimeConfig:"file_manager=(close_handle_minimum=100,close_idle_time=300,close_scan_interval=30)"

    Description

      Periodic drops in throughput can be observed during checkpoints while running with an artificial workload and specific WiredTiger tuning parameters enabled.

      I've attached diagnostic.tar.gz that shows this behavior. At around 2017-10-20T20:10:08.587Z the following command was executed:

      db.adminCommand({setParameter: 1, wiredTigerEngineRuntimeConfig:"file_manager=(close_handle_minimum=100,close_idle_time=300,close_scan_interval=30)"});

      After reaching steady state, we see the following:

      Additionally, from perf, we can see that after making this parameter change, a single cpu core (12.5% of the total) is consistently fully utilized running eviction: 

      start_thread+0xffff017a586c80c5
      		 
      __wt_thread_run+0xffff5458d1e2e016
      		 
      __wt_evict_thread_run+0xffff5458d1e2e951

      Attachments

        1. checkpoint_lock_4.2.7.png
          checkpoint_lock_4.2.7.png
          176 kB
        2. diagnostic.tar.gz
          69.76 MB
        3. mix.js
          6 kB
        4. no_checkpoint.png
          no_checkpoint.png
          209 kB
        5. periodic_drops.png
          periodic_drops.png
          339 kB

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. WT-5479 Checkpoint schema lock can cause long stalls in command execution on systems with many active data handles

          • Major - P3 - Major loss of function.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. SERVER-40773 Slow queries every 80s under moderate load

          • Major - P3 - Major loss of function.
          • Closed
          related to

          Bug - A problem which impairs or prevents the functions of the product. SERVER-32890 Background index creation sometimes block whole server for dozen of seconds (even with empty collections)

          • Major - P3 - Major loss of function.
          • Closed

          Task - A task that needs to be done. WT-6421 Avoid parsing metadata checkpoint for clean files

          • Major - P3 - Major loss of function.
          • Closed

          Activity

            People

              brian.lane@mongodb.com Brian Lane
              kelsey.schubert@mongodb.com Kelsey Schubert
              Votes:
              13 Vote for this issue
              Watchers:
              41 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: