Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-4190

Decrease shutdown time by doing multi-threaded cache flush

    • Type: Icon: Improvement Improvement
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • WT3.2.1, 4.3.1, 4.2.0-rc1
    • Affects Version/s: 3.6.5, 4.0.0
    • Component/s: None
    • Environment:
      Linux, CentOS7, 256gb RAM, fast SSD, 56 cores
    • 1
    • Storage NYC 2019-03-25, Storage Engines 2019-06-03
    • v4.2

      I have been running the insert benchmark for MongoDB 3.6.5 and 4.0.0 with WiredTiger. Some of what I learned is at https://groups.google.com/forum/#!topic/wiredtiger-users/FvVMCVqLa_E

      This is open to discuss making mongod shutdown faster. Here is one example where each line represents a separate configuration. The legend is:

      • secs - seconds to shutdown
      • cacheGB - size of WT cache in GB
      • sps - syncPeriodSecs value
      • ct - value of eviction_checkpoint_target, def == default
      • dirty - value of eviction_dirty_trigger, def == default
      • config - see gist
      • wGB - GB of writes at shutdown, per iostat
      • wMB/s - average rate for write MB/s at shutdown

      secs    cacheGB sps     ct      dirty   config  wGB     wMB/s

       63     120      60     def     def     v1       8.0    135.5

       65     120     300     def     def     v2       8.1    133.0

      789     120      60      1      90      v6      43.5     55.2

      816     120      60     10      90      v7      43.5     53.4

      476     120     300      1      90      v10     27.6     58.1

      471     120     300     10      90      v11     27.8     59.1

      123     120     300      0      20      v13     19.3    157.1

      120     120     300      1      20      v14     12.3    103.1

      315     120     300      0      40      v16     22.5     71.7

      308     120     300      1      40      v17     21.1     68.8

      357     120     300      0      60      v19     17.4     48.8

      354     120     300      1      60      v20     16.5     46.7

      448     120     300      0      80      v22     23.8     53.0

      447     120     300      1      80      v23     24.0     53.7

      {{ 68     120      30     def     def     v25     18.9    281.8}}

      {{ 65     120     180     def     def     v26     22.0    343.3}}

      {{ 64     120     600     def     def     v27     32.5    516.5}}

      The min time to shutdown is 60 seconds. I get that a b-tree needs to flush dirty pages at shutdown or do crash recovery at startup so it can't be instantaneous. But from the wMB/s numbers above, except for the last 3 configurations, the write IO rate isn't great considering the storage devices can sustain more than 1GB/s.

      This was measured by doing the load for the insert benchmark with 250m docs inserted by 16 clients, each using a separate collection and then shutting down mongod when the insert ends.

      See http://smalldatum.blogspot.com/2017/06/the-insert-benchmark.html for more on the insert benchmark and that includes a link to the code.

            Assignee:
            alex.cameron@mongodb.com Alex Cameron (Inactive)
            Reporter:
            mdcallag Mark Callaghan
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated:
              Resolved: