Background

The metrics class expects only one resharding operation node type to be active at once for a particular process. This is a problem on processes that have both donors and recipients. It causes issues with both double-starting and the “quicker” node type ending the operation before the “slower” node type is finished.

Proposed Solution

1. Change the participant state enums to be boost::optionals
2. onStart will emplace a state enum with kUnused***
3. onStart will check for an emplaced donor/recipient and allow no-op if the other type is already emplaced. If any other combination of enum types is placed down, it will trip an invariant.
4. onCompletion/onStepDown will only remove the enum for that particular node type, letting the last node type to quit truly shut down the metrics system.
5. onCompletion/onStepDown will only take its expected actions if it is the last node type existing on that process for the resharding operation.

***You’d think that we can just increment the state past kUnused to indicate that a donor/recipient is active on the process, but the way metrics are written, the state can’t be incremented until it’s been persisted on disk. The metrics system is started before any state changes are persisted on disk, so making the state enums optional is our way of satisfying the state transition ordering.