Problem

checkDurableShardCatalog() in src/mongo/db/shard_role/shard_catalog/metadata_consistency_checks/database_metadata_checks.cpp runs unguarded on delayed secondaries, comparing a lagged local read of the shard catalog against a fresh view of the global catalog. Any sharding DDL in flight during the check is therefore reported as a metadata inconsistency.

This is the root cause of BF-46010, where CheckMetadataConsistencyInBackground failed three unrelated tests in //jstests/suites/cluster-scalability:sharding_jscore_passthrough_last_lts_new_old_old_new.

Details

The shard catalog (config.shard.catalog.databases) is ordinary replicated data within a shard's replica set. readDatabaseFromDurableShardCatalog() reads it with a DBDirectClient, i.e. locally on whichever node executes the check. Secondaries participate via _shardsvrCheckMetadataConsistencySecondaryParticipant.

There are two secondary modes (shardsvr_check_metadata_consistency_participant_command.cpp:147-165):

• kCheckAtPrimaryTimestamp sets afterClusterTime at the majority commit time, so the secondary waits until it has caught up. Safe.
• kCheckAtSecondaryTimestamp sets no readConcern and signals the secondary via RSNodeMode::kDelayedSecondary that it may be lagged.

In checkDatabaseMetadataConsistency(), kDelayedSecondary only guards the in-memory cache check:

// line 249 - unguarded
std::vector[MetadataInconsistencyItem] inconsistencies = checkDurableShardCatalog(opCtx, checkCtx);

// lines 253-257 - guarded
// There is currently no way to retrieve a DSR from a given timestamp, so we skip this check on
// delayed secondaries.
// TODO (SERVER-130947): maybe you can.
>;if (checkCtx.rsMode != RSNodeMode::kDelayedSecondary) {
    auto cacheInconsistencies = checkShardCatalogCache(opCtx, checkCtx);
    ...
}

The durable check has no equivalent guard.