Context

FCBIS works by copying WT files from its sync source to a local temp directory called .initialsync, then moving these files to the original dbPath that the node is running on. In order to perform this move, FCBIS must restart the storage engine because we cannot safely move or delete WT files while they are in use. Each time FCBIS restarts and switches WT to a different directory, it verifies the last shutdown state of that directory is Clean. This is testing the directory that we are switching to. Prior to v8.3, each storage engine reinit created a new lock file in that directory, so the check reflected the directory that was being opened, not the original dbPath directory.

This changed in SERVER-110717, which moved lock file creation from the storage engine to mongod_main as a decoration on the service context (process level). As a result, when we switch directories in FCBIS to the new directory, we still fetch the process-level last shutdown state, which states that we are in unclean shutdown, crashing the node. This is likely unrecoverable unless we clear the dbPath and restart FCBIS from the beginning, as each subsequent restart will run into the exact same error.

Acceptance Criteria

Ensure that the invariant matches what we expect the lock file behavior should be. The two most likely solutions are either relaxing the invariant if it is no longer providing value, or modifying the way storage engines reports on the lock file if we still would like to keep that invariant.

This ticket will need to be backported to v9.0 and v8.3