A background thread should be added as a resmoke.py hook (i.e. a class to run dedicated logic before a test starts or after a test finishes) to control whether failovers are being triggered on the resmoke.py fixture. We don't want a stepdown to occur while we are validating data consistency (i.e. during ValidateCollections and CheckReplDBHash), so being able to suspend the stepdown thread is a necessary feature.
- Create buildscripts/resmokelib/testing/hooks/stepdown.py file.
- Define a _StepdownThread class as a subclass of threading.Thread with a run() method that behaves roughly as follows:
Note: In addition to a "running" and "terminated" state, the StepdownThread could have an explicit "suspended" state where it waits on a condition variable until signaled by resmoke.py's job thread to resume triggering failovers. This would avoid continuously spawning and joining the stepdown thread every time the resmoke.py hook runs.
- Define a ContinuousStepdown class as a subclass of the hooks.interface.CustomBehavior class with a before_test() method that spawns a _StepdownThread instance (or signals it via a condition variable to resume), and an after_test() method to suspend the thread. The configuration options for the ContinuousStepdown class should match what is mentioned in