Some query stages implement spilling by writing to a WT table. If this happens inside the context of a user multi-doc transaction, these writes will be included in the same storage engine transaction. As uncommitted writes are not evictable to disk by WiredTiger, this has the consequence of filling up WiredTiger's dirty cache with un-evictable data.

This has a few problems:

• Spilling in a multi-doc transaction will create negative performance consequences and potentially stall any concurrent write workload, as we can pollute the WT cache with un-evictable data
• Because no spilling is happening, we actually are still imposing a limit based on the size of the WT cache, meaning that user queries will fail if they spill enough data.
• We're running into WiredTiger bugs because we drop the tables before the transactions commit/abort (WT-10576)

I see two solutions:

• We can ban spilling in multi-doc transactions until we come up with a better solution
• Spill in a separate storage transaction. We have to be incredibly careful here because this can cause deadlocks (see SERVER-61116), and would need close collaboration with Storage Execution. To make this work, we would need to do something similar to what we did for SERVER-62650. In the spilling transaction, we would always need to impose a timeout on how long it will block on cache eviction. Otherwise, we have to force the user's transaction to roll-back.