We have encountered numerous instances of performance issues resulting from a collision of io requests from checkpoints and journal writes.

The symptoms are a known performance issue resulting from placing the dbPath directory and the WiredTiger journal directory on the same filesystem. The bursts of IO to WiredTiger datafiles that occur at the end of each checkpoint when filesystem buffers are flushed to disk can saturate the disk, delaying writes to the journal files, and producing small IO "stalls" exactly like those observed on the mongod logfiles.

The remedy to this problem is relatively simple – to place the mongod dbPath directory and the mongod dbPath/journal directory on separate physical devices and use a symlink to point to the journal directory from the mongod dbPath.

Physically separating the IO removes the contention between database files and journal files during checkpoints, and prevents the small IO stalls observed in this case and others like it.