I'm running a 3 member replicaset. The average workload is very constant and does not push load average above 2-3. When workload increases (for example due to a delay in our processing queue) mongodb starts processing queries very fast but after about 5-10 minutes completely starts hogging userspace CPU usage (100%, load average 120-130).

Even when completely shutting down the workload, load average still stays at 120-130, the only way for it to go back to normal is running rs.stepDown() and force another member to be primary. This instance then (usually) starts processing the requests very fast but sometimes still goes back to the very high load average.

In the mongodb log I can see write requests are taking a very long time (some upto 10 seconds).

I'm not really sure how to prevent this as it completely disables my whole workload.