On sharded clusters, the collMod DDL coordinator first sends the collMod participant command to the primary shard, then broadcast it to all other shards. If the responses to the broadcast are retriable errors, the coordinator will retry, and eventually collMod will complete successfully (see: SERVER-90117).

However, if one of the responses is a non-retriable error, the coordinator will not retry, but rather give up and fail. Since the collMod was executed on the primary shard but not the other shards, the collection options became inconsistent across shards, as reported by checkMetadataInconsistency.

One instance where this happens is when an index build is in progress in a non-primary shard. That shard will return a BackgroundOperationInProgressForNamespace error, which is not retriable, and collMod will fail midway leaving the collection options inconsistent across shards (see attached repro).