During MongoDBs development cycle, a binary built from master allows upgrading to the "latest" FCV, e.g. binary v9.0 can be upgraded to FCV 9.0. However that upgrade is only 'nominal' because FCV 9.0 is still under development and further persisted metadata format changes will land until FCV 9.0 stabilizes when the v9.0 release branch is cut.

So a cluster that underwent a 'nominal' upgrade (due to using a master development binary) silently reports FCV 9.0 but it will be missing some metadata format changes i.e. some data will remain in old 8.x formats. It is likely that once MongoDB 9.1 binaries are swapped in, which remove support for pre-FCV 9.0 formats, the cluster will behave unexpectedly (e.g. crash/data loss).

Add a guardrail preventing upgrades to a non stable FCV outside testing, e.g.:

• Add a lastStableFCV field to releases.yml, that will be e.g. FCV 9.0 for the entire v9.1 development cycle.
• The Server binary only allows upgrades to a FCV greater than lastStableFCV if started with --enableTestCommands.
• Document that the lastStableFCV field should be updated when no new format changes can be done (likely only right before the release becomes GA).

We must also e.g. adjust the cluster initiation logic so a new non-test replica set is on lastStableFCV, not latestFCV.