This ticket tracks a potential future scalability issue with WiredTiger metadata and establishes reducing its footprint as a design goal.

Metadata is required for disaggregated checkpoint pickup (WT-16188), logical-size accounting (WT-18079), checkpoint processing, recovery, and schema operations. The metadata btree is effectively cache-resident. Its size therefore represents both persistent-storage and cache-memory overhead.

Scaling concern

MongoDB creates a WiredTiger table for every collection and index. Each ident creates three metadata records in classic storage. Disaggregated storage creates five local records and four shared records.

Each record currently contains a fully resolved configuration string, including defaults repeated across every ident. Measurements with MongoDB-like ident names show this cost scaling with ident count:

Initial recommendation

A proof of concept omitted configuration values equal to WiredTiger defaults from persisted metadata. At 10K idents, this reduced disaggregated metadata from 64.26 MiB to 26.16 MiB, approximately a 2.5× reduction.

This validates default-value exclusion as one possible improvement, but it should not constrain the eventual design.

Further investigation

The next step here would be to understand from the customer/product perspective what a realistic number of tables is in year to come, with that in hand a spike should evaluate additional approaches for reducing metadata further. Methods may include dictionary compression for repeated keys and values. It should also consider alternative encodings and opportunities to reduce duplication between local and shared metadata.

The intended outcome is a recommended long-term representation that minimizes the cache and storage cost of metadata at high ident counts.