MDB currently has powercycle and process termination tests which have historically discovered durability bugs. It's not obvious those are fundamentally insufficient, but we've found other durability bugs when using other techniques. Specifically we can set up a mongodb cluster and:
- Run a workload against the cluster
- SIGSTOP one of the mongodb processes
- Copy that process' dbpath to a tmp path with direct I/O
- SIGCONT the paused process
- Start a mongod on the tmp path
- Run validate on all collections in the tmp path
- Repeat in a tight loop.
There's interest in permanently adding this to our testing suites. Attached is an unrefined (apologies) patch that can be used as a starting point in implementing the above.