-
Type:
Improvement
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Storage Engines, Storage Engines - Persistence
-
336.678
-
SE Persistence backlog
-
None
Issue Summary
During disaggregated-storage checkpoint pick-up on a follower with many tables, 75.4% of the pick-up time is spent assembling file metadata configuration in *create_file: 13,669 ms of an 18.1 s apply for 50,029 new layered tables (BF-44421 investigation). Two operations make up that cost: *wt_config_collapse and __wt_config_tiered_strip. This ticket covers the strip half: skip it entirely when the connection has no tiered storage, because the only thing it strips is tiered_storage.shared, which can never be set on such a connection. Expected saving on the measured workload: ~5 s of startup.
Context
- Measured on disagg_measure_startup_primary_25k_locust (BF-44421): startup checkpoint pick-up applies 50,029 new layered tables; each creates an ingest table through *create_file, which runs *wt_config_collapse then __wt_config_tiered_strip on a stack of ~50-key config strings. Per-table cost in-workload: 273 us; 50,029 tables.
- Local micro-benchmark replaying the exact cfg stack (file_meta base + ingest template + id/version string, Release build, 20k iterations): collapse 65.2 us/call (62%), strip 39.2 us/call (38%).
- Profile of the strip (__wt_config_merge): ~40% per-entry vsnprintf formatting of every key and value, ~18% malloc/free (~280 pairs per call), ~7% qsort strcmp, ~19% output formatting. Linear work, but heavy.
- Disaggregated storage never configures tiered storage: WT_CONN_TIERED_STORAGE_ENABLED (conn->bstorage != NULL) is only set when a named storage source is configured.
- Complementary prior work: WT-18174 removed collapse from the checkpoint-metadata update path; WT-18472 (done) removed 94% of the cursor-merge cost with prefetch; WT-17779 covers the stub-file creation (12.8% of pick-up). The collapse half of the config cost is handled separately.
Proposed change
Guard the strip at the __create_file call site:
/\*
\* Strip any configuration settings that should not be persisted. Only tiered storage
\* settings are stripped; there is nothing to do unless the connection uses tiered storage.
\*/
if \(WT\_CONN\_TIERED\_STORAGE\_ENABLED\(S2C\(session\)\)\) {
filecfg\[1\] = fileconf;
filecfg\[2\] = NULL;
WT\_ERR\(\_\_wt\_config\_tiered\_strip\(session, filecfg, &filestripped\)\);
} else {
filestripped = fileconf;
fileconf = NULL;
}
Call-site placement is deliberate. An in-function guard would silently switch every caller from merge semantics (additive across strings, nested-struct merging) to collapse semantics (output masked to cfg[0]'s keys). All four current callers pass a full generated base as cfg[0] so it would be safe today, but nothing enforces that precondition for future callers. It would also needlessly change the turtle file's bytes (via *metadata_config) and re-collapse an already-collapsed string in the hot path. The two tiered-only callers (*tiered_metadata_insert, __create_tiered) never fire the guard either way.
Why this is safe
- The strip list is a single key: tiered_storage=(shared=). On a non-tiered connection, shared can only ever be the default false — table-level tiered storage requires connection-level tiered storage, enforced in __wti_tiered_bucket_config.
- Byte differences in persisted file: metadata are limited to default-valued keys, verified byte-for-byte locally: tiered_storage=(...) keeps shared=false, and user-specified nested structs (e.g. disaggregated=(page_log=none,storage_source=none)) no longer inherit default-valued sub-keys from the base (e.g. storage_tier=none). Both forms read back identically through the config API.
- Key order is unchanged: the generated base is already alphabetical, matching the merge's sorted output.
- Local validation: full catch2 suite; Python suite subsets covering config parsing, schema create, metadata cursors, dump, tiered (strip still active when tiered is enabled), and all test_layered_schema* including the pick-up reproducer — all green.
When tiered storage is removed
- This block reduces to the else branch; simplify further to WT_ERR(__wt_metadata_insert(session, uri, fileconf)); and drop filestripped.
- __wt_config_tiered_strip is deleted along with tiered storage; its remaining callers convert to plain collapse.
- The metadata byte difference disappears on its own: once the generated bases no longer contain tiered_storage, there is no shared=false left to retain.
Definition of done
- Guard merged at the __create_file call site.
- Instrumented Evergreen run of the many-collections disagg startup workload showing the config-assembly phase statistic reduced by ~35-40% (~5 s off startup pick-up).
- Full test suite green in CI.
- related to
-
WT-18517 Make __wt_config_collapse linear in the configuration size
-
- Needs Scheduling
-