In order to ensure FCV stability without holding the FCV lock during expensive operations, it will need to be implemented a component enclosing the following logic:
- Take FCV lock in Shared mode
- Quickly check what is the current FCV and make up some decision about what to do (return or go to step 3)
- Write some Placeholder somewhere (most likely this will be the Primary Only Service) to indicate that there is an operation running under a specific FCV version
- Release the FCV lock
- Do the expensive/network communicating work outside of any locks and possibly on a different thread, etc...
- Remove the Placeholder from (3)
The Placeholder from (3), which will be used to decide whether it's allowed to [down|up]grade the FCV: it will be needed to wait for all of the Placeholder(s) to drain before setting the FCV to a different version.