From HELP-93859, the customer ran into a situation where /data ran out of disk space while building indexes. 

Summary

Index build validation is a feature that was introduced with v8.1 going forward. This feature checks index metadata before transitioning to the Applying phase as is intended to address this class of issue, where the recipient collection is missing expected indexes.

Root Cause

In v8.0 and earlier versions, it is possible to experience this behavior, where indexes may be missing despite a successful resharding status. Note that this is also possible in later versions if the feature is disabled by default or by customer override.

During resharding, an index build is initiated with a startIndexBuild oplog entry, which is replicated to all secondaries.

The likely scenario occurred as such in _buildIndexThenTransitionToApplying():

1. NodeA (recipient primary) runs out of disk space; the index build is killed and NodeA writes an abortIndexBuild oplog entry before stepping down
2. NodeB steps up as new primary before it has applied NodeA's abortIndexBuild, but it has already applied NodeA's startIndexBuild, so the build is still tracked as active
3. NodeB's resharding service restarts the index build phase; startIndexBuild() returns IndexBuildAlreadyInProgress because the old build is still registered
4. NodeB waits for the in-progress build to clear, the abortIndexBuild oplog arrives and cleans it up, and resharding transitions to kApplying with no indexes built

Fix

Since the resharding process is able to transition past the building-indexes phase when indexes are missing, this represents an idempotency issue and needs to be addressed in a more formal way.

This ticket will involve coming up with a more formal way to signal that indexes have successfully built in failover scenarios. Currently, we only track if the index build has "finished", regardless of whether it was successful or aborted/failed.