Background

SPM-4023 - Symmetric FCV Transition changed setFCV semantics where an FCV transition can still be happening but the in-memory FCV state evaluates as if the FCV is fully upgraded/downgraded and not in a transitional state.

Bug

• This causes a subtle bug with resharding:
  • When config server is in the enableTargetFeatures set fcv phase, it's in-memory FCV acts as if FCV is fully upgraded/downgraded (here).
  • As a result, the isUpgardingOrDowngrading check that prevents resharding from starting during FCV upgrade/downgrade doesn't trigger (here).
  • Resharding is then allowed to start (after setFCV aborted all reshardings here).
  • Since config server only transitions participants to enableTargetFeatures , phase after transitioning itself (here), it's possible that a participant shard can be temporarily acting on an older FCV, before later acting on a new oner, which may cause inconsistencies.
• Before SPM-4023, it was not like this. The config server used to only set it's FCV to be the fully upgraded/downgraded state after all shards fully upgrade/downgraded their FCV.

Solution
Called with joan.bruguera-mico@mongodb.com  and randolph@mongodb.com to discuss and here was our conclusion:

• Short term solution (this ticket): don't allow resharding to start at any point during FCV transition (adding a check for the presence of phase in the fcv doc before resharding starts).
  • This eliminates the edge case where coordinator is on 9.0 but some shards are still in kUpgrading (which evaluates to 8.0)
    • Although we think this edge case is theoretically safe — the operation's OFCV is forwarded to participants and lives on the opCtx — we can't guarantee it today, since code we don't own can spawn a new opCtx that doesn't carry the OFCV.
  • This also eliminates the edge case where coordinator is on 8.0 but some shards are still in kDowngrading , and there is no OFCV or pinned FCV that can be used, which triggerd the tassert in BF-44533.
    • As an alternate solution, we discussed adding an extra case to getVersionContextOrDefault, where if there is no pinned fcv and global FCV is kDowngrading, then we use the target downgrading FCV (8.0/8.3) as the default. But we decided not to do this because it adds another default case whose correctness is not straightforward to reason about.

• Long term solution:it's not ideal that resharding has to know about FCV internals (checking the phase field). In the future it shouldn't matter that resharding starts during an FCV transition, because the OFCV would be forwarded and used consistently across all shards. But right now that's hard to guarantee because, as stated above, code we don't own can create a new opCtx that won't use the OFCV.
• We also discussed the potential for resharding to be aborted and drained before FCV is set to transitional state so we don't have to worry about resharding running at all during setFCV.