The dual catalog cache workstream was a necessary technical step to enable the transition to authoritative shards. Although this step is required, it doubles the memory footprint of the CatalogCache in the running MongoD process.

The original plan was to introduce the dual catalog cache as a binary change, and then, during the FCV upgrade, shift to relying on authoritative shards. This would eliminate the need to use the CatalogCache for filtering metadata, thereby reducing its memory footprint back to its original size.

The problem arises if we want to make SPM-3729 production-ready without delivering collection-level authoritative shards. In that case, we still need the dual catalog cache, but it would result in an increased memory footprint for collection metadata throughout the entire 9.0 FCV lifecycle.

Proposed workaround:

• Database metadata: introduce a new feature flag gDatabaseDualCatalogCache
  • This flag will enable a second catalog cache instance, used exclusively for database metadata filtering usages.
  • When enabled, it will configure the Grid::CatalogCache to use the ConfigServerCatalogCacheLoader for database metadata routing usages, fully bypassing SSCCL for this path.
• Collection metadata: no change of behavior from 8.0
  • Continue to rely on the existing Grid::CatalogCache and ShardServerCatalogCacheLoader for collection metadata routing and filtering usages, for now.

Future step:
Enable the existing gDualCatalogCache flag once shards are ready to become authoritative for collection metadata. This will make collection metadata filtering usages rely on the ShardServerCatalogCacheLoader and routing usages on the ConfigServerCatalogCacheLoader.