Overview

setFCV must force a stable checkpoint after majority-committing the FCV document update and wait until lastStableRecoveryTimestamp is past the FCV change opTime before returning. Until this is guaranteed, swapping binaries immediately after setFCV completes can result in crashes on node restart.

Background

During FCV upgrade/downgrade, setFCV:

1. Majority-commits the FCV document (kUpgrading/kDowngrading to target FCV)
2. Drains in-flight operations
3. Majority-commits the final FCV document confirming the transition

Majority write concern advances the in-memory stable timestamp but does not guarantee a WiredTiger stable checkpoint has been taken at that timestamp. lastStableRecoveryTimestamp only advances after WiredTiger completes a stable checkpoint at the new stable ts.

If a binary swap happens after setFCV returns but before a checkpoint is taken past the FCV change opTime, a node restart will:

1. Recover from its last stable checkpoint (which predates the FCV change)
2. Re-apply oplog entries from the FCV change opTime onwards (Might also results in data loss).
3. Encounter oplog entries that reference the new FCV version or binary semantics
4. Crash — the old binary cannot parse entries generated under the new FCV

This was surfaced while investigating SERVER-129597 (initial sync not resilient to FCV changes), which is a specific instance of this broader gap.