We've added a new option to checkMetadataConsistency called performStrictChunkChecksIfBelowThreshold. This option by default is set to 1152921504606846975 = 2^60 - 1.

What that parameter does is enable a more comprehensive set of chunk checks if the number of chunks present on the shard is below a threshold. These new checks are only enabled on FCV 9.0.
If the number of chunks goes above the threshold then the checks are disabled in favour of just checking that the number of chunks matches as well as the shard version.

Additionally, new metrics have been added to the shardingStatistics section in serverStatus:
1. numberOfInconsistenciesFound: tracks the number of inconsistencies
found
2. numberOfDatabasesChecked: tracks the number of databases checked in
checkMetadataConsistency calls
3. numberOfCollectionsChecked: tracks the number of collections checked
in checkMetadataConsistency calls
4. numberOfChunksChecked: tracks the number of chunks checked in
checkMetadataConsistency calls
5. activeDdlLocksHeldForDatabase: number of active DDL locks held for
databases in checkMetadataConsistency calls
6. activeDdlLocksHeldForDatabaseDurationMillis: running total time the
DDL lock for databases has been held in checkMetadataConsistency calls
7. activeDdlLocksHeldForCollection: number of active DDL locks held for
collections in checkMetadataConsistency calls
8. activeDdlLocksHeldForCollectionDurationMillis: running total time the
DDL lock for collections has been held in checkMetadataConsistency calls
9. ddlLockHeldForDatabaseDurationMillis: cumulative total time the DDL
lock for databases has been held in checkMetadataConsistency calls
10. ddlLockHeldForCollectionDurationMillis: cumulative total time the
DDL lock for collections has been held in checkMetadataConsistency calls

Description of Linked Ticket

checkMetadataConsistency has gotten bigger with shards authoritative. Specifically, we are checking chunk metadata between the CSRS, shards, and the in-memory chunk map - which we did not do before. This is done under a DDL lock, preventing DDLs on the namespace. Also, this might lead to reading a lot more content from the CSRS.

We should do at least the following:

1. Build observability in the impact from CMC: lock timing/contention, chunk and collection count, etc
2. Offer a flag as a means to not do a full chunk information compare. Continue to do a soft check like the count, version, etc.
3. Offer a knob that can dial a ful check on/off - example, if less than 10k chunks, do a full check otherwise only a soft check