Performing the entire spill under a single lock interacts very poorly with flow control. We trigger a replication lag spike because of the large volume of writes, which activates flow control. Because we do all of the work under a single lock, flow control throttles the user operation but not the index build, which is the exact opposite of what we want. In addition, doing 50 million document insertions under a single lock makes flow control immediately scale down to the minimum number of locks/second issued, and then it only slowly ramps back up the correct value.

To fix this, we should yield locks in between each batch that we write. Initially we'll only do this for normal spills during the collection scan and not spill merging, as the latter is both more complicated and rarer.

Benchmarking shows that this improves foreground workload performance by 20% at the cost of making the index build itself take 6% longer. This is a tradeoff we're currently happy to make.