[SERVER-81037] Cluster the local durable catalog or index it by namespace Created: 13/Sep/23  Updated: 16/Oct/23

Status: Backlog
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Josef Ahmad Assignee: Backlog - Catalog and Routing
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-46105 Allow new collection creation inside ... Backlog
is depended on by SERVER-82235 Make logical initial sync use $listCa... Backlog
Assigned Teams:
Storage Execution EMEA
Participants:

 Description   

Today, the local catalog implements some level of MVCC outside of the storage engine to detect conflicting DDL operations – e.g. concurrent creations of the same namespace. We should be able to revisit the durable catalog schema to push this responsibility back to the storage engine. This has many benefits: it simplifies the catalog, enables new use cases natively (e.g. support for prepare conflicts), allows for more efficient point-in-time lookups of durable catalog data, and is a step towards a durably consistent catalog between replicas.
 
Clustering the catalog with a cluster key on the "ns" field enables collection creation inside prepared transactions (SERVER-46105), which would otherwise require supporting prepared transactions and prepare conflicts above the storage engine.
An alternative approach would be a unique index on the "ns" field. This is more involved and would entail promoting the catalog to a regular collection, which is a direction we may want to take longer-term.



 Comments   
Comment by Josef Ahmad [ 05/Oct/23 ]

This will also make $listCatalog lookups efficient, as filtering by namespace or UUID will happen at the source.

Generated at Thu Feb 08 06:45:17 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.