While gradually rolling out background compaction (autoCompact) on one of our production clusters, we observed what appears to be a severe and sustained performance degradation. After autoCompact had been enabled for some time, application QPS dropped sharply and stayed low for an extended period, and the WiredTiger cache dirty ratio remained at around 20%. The cluster did not seem to recover on its own; once autoCompact was disabled, the dirty ratio came down and QPS recovered.

We were subsequently able to reproduce this offline. Our current understanding is that autoCompact keeps the data file in first-fit allocation mode for long periods, and that __block_first_srch() performs an O linear scan over a very large avail extent list while holding block->live_lock. As the scan gets longer, block allocation slows down, dirty pages cannot be written out quickly enough, and we end up with the dirty-cache backpressure and sustained QPS drop seen in production.

We would be very grateful for your thoughts on this analysis, and we are of course happy to be corrected if we have misread any part of the code.