Decrease shutdown time by doing multi-threaded cache flush

I have been running the insert benchmark for MongoDB 3.6.5 and 4.0.0 with WiredTiger. Some of what I learned is at https://groups.google.com/forum/#!topic/wiredtiger-users/FvVMCVqLa_E

This is open to discuss making mongod shutdown faster. Here is one example where each line represents a separate configuration. The legend is:

• secs - seconds to shutdown
• cacheGB - size of WT cache in GB
• sps - syncPeriodSecs value
• ct - value of eviction_checkpoint_target, def == default
• dirty - value of eviction_dirty_trigger, def == default
• config - see gist
• wGB - GB of writes at shutdown, per iostat
• wMB/s - average rate for write MB/s at shutdown

secs    cacheGB sps     ct      dirty   config  wGB     wMB/s

 63     120      60     def     def     v1       8.0    135.5

 65     120     300     def     def     v2       8.1    133.0

789     120      60      1      90      v6      43.5     55.2

816     120      60     10      90      v7      43.5     53.4

476     120     300      1      90      v10     27.6     58.1

471     120     300     10      90      v11     27.8     59.1

123     120     300      0      20      v13     19.3    157.1

120     120     300      1      20      v14     12.3    103.1

315     120     300      0      40      v16     22.5     71.7

308     120     300      1      40      v17     21.1     68.8

357     120     300      0      60      v19     17.4     48.8

354     120     300      1      60      v20     16.5     46.7

448     120     300      0      80      v22     23.8     53.0

447     120     300      1      80      v23     24.0     53.7

{{ 68     120      30     def     def     v25     18.9    281.8}}

{{ 65     120     180     def     def     v26     22.0    343.3}}

{{ 64     120     600     def     def     v27     32.5    516.5}}

The min time to shutdown is 60 seconds. I get that a b-tree needs to flush dirty pages at shutdown or do crash recovery at startup so it can't be instantaneous. But from the wMB/s numbers above, except for the last 3 configurations, the write IO rate isn't great considering the storage devices can sustain more than 1GB/s.

This was measured by doing the load for the insert benchmark with 250m docs inserted by 16 clients, each using a separate collection and then shutting down mongod when the insert ends.

See http://smalldatum.blogspot.com/2017/06/the-insert-benchmark.html for more on the insert benchmark and that includes a link to the code.