Background

A v4.0 instance running as part of our Evergreen infrastructure stalled for 10 minutes without accepting reads. There were a few slow queries that got killed by the Cloud Manager around this time though it's not clear whether this is related.

Most of the abnormalities in the FTDC seem to stem from the fact that eviction is ineffective during this period due to hazard pointers blocking eviction. Checkpoint takes longer around this time but since eviction isn't blocked by the checkpoint (and is instead blocked by hazard pointer acquisitions), I'm led to believe that the ineffectiveness of eviction is slowing the checkpoint down since it can't read pages into cache rather than the other way around which is normally the case (checkpoint disrupting eviction).

In the original HELP ticket, we speculated a bit on the operations that got killed (they are aggregations which can be expensive) but regardless of that, it's not clear why this would involve hazard pointers being held for extended periods of time.

Some more preliminary investigation can be found on the linked HELP ticket but I've attached the FTDC, logs and relevant screenshot on this ticket.

Goal

We should aim to identify what the problem is and if the change is trivial, make a fix. If it's non-trivial, we should create another follow-up ticket to implement this change.