Skip tiered-storage config strip during file creation when tiered storage is disabled

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Storage Engines, Storage Engines - Persistence
    • 336.678
    • SE Persistence backlog
    • None

      Issue Summary

      During disaggregated-storage checkpoint pick-up on a follower with many tables, 75.4% of the pick-up time is spent assembling file metadata configuration in *create_file: 13,669 ms of an 18.1 s apply for 50,029 new layered tables (BF-44421 investigation). Two operations make up that cost: *wt_config_collapse and __wt_config_tiered_strip. This ticket covers the strip half: skip it entirely when the connection has no tiered storage, because the only thing it strips is tiered_storage.shared, which can never be set on such a connection. Expected saving on the measured workload: ~5 s of startup.

      Context

      • Measured on disagg_measure_startup_primary_25k_locust (BF-44421): startup checkpoint pick-up applies 50,029 new layered tables; each creates an ingest table through *create_file, which runs *wt_config_collapse then __wt_config_tiered_strip on a stack of ~50-key config strings. Per-table cost in-workload: 273 us; 50,029 tables.
      • Local micro-benchmark replaying the exact cfg stack (file_meta base + ingest template + id/version string, Release build, 20k iterations): collapse 65.2 us/call (62%), strip 39.2 us/call (38%).
      • Profile of the strip (__wt_config_merge): ~40% per-entry vsnprintf formatting of every key and value, ~18% malloc/free (~280 pairs per call), ~7% qsort strcmp, ~19% output formatting. Linear work, but heavy.
      • Disaggregated storage never configures tiered storage: WT_CONN_TIERED_STORAGE_ENABLED (conn->bstorage != NULL) is only set when a named storage source is configured.
      • Complementary prior work: WT-18174 removed collapse from the checkpoint-metadata update path; WT-18472 (done) removed 94% of the cursor-merge cost with prefetch; WT-17779 covers the stub-file creation (12.8% of pick-up). The collapse half of the config cost is handled separately.

      Proposed change

      Guard the strip at the __create_file call site:

              /\*
               \* Strip any configuration settings that should not be persisted. Only tiered storage
               \* settings are stripped; there is nothing to do unless the connection uses tiered storage.
               \*/
              if \(WT\_CONN\_TIERED\_STORAGE\_ENABLED\(S2C\(session\)\)\) {
                  filecfg\[1\] = fileconf;
                  filecfg\[2\] = NULL;
                  WT\_ERR\(\_\_wt\_config\_tiered\_strip\(session, filecfg, &filestripped\)\);
              } else {
                  filestripped = fileconf;
                  fileconf = NULL;
              }
      

      Call-site placement is deliberate. An in-function guard would silently switch every caller from merge semantics (additive across strings, nested-struct merging) to collapse semantics (output masked to cfg[0]'s keys). All four current callers pass a full generated base as cfg[0] so it would be safe today, but nothing enforces that precondition for future callers. It would also needlessly change the turtle file's bytes (via *metadata_config) and re-collapse an already-collapsed string in the hot path. The two tiered-only callers (*tiered_metadata_insert, __create_tiered) never fire the guard either way.

      Why this is safe

      • The strip list is a single key: tiered_storage=(shared=). On a non-tiered connection, shared can only ever be the default false — table-level tiered storage requires connection-level tiered storage, enforced in __wti_tiered_bucket_config.
      • Byte differences in persisted file: metadata are limited to default-valued keys, verified byte-for-byte locally: tiered_storage=(...) keeps shared=false, and user-specified nested structs (e.g. disaggregated=(page_log=none,storage_source=none)) no longer inherit default-valued sub-keys from the base (e.g. storage_tier=none). Both forms read back identically through the config API.
      • Key order is unchanged: the generated base is already alphabetical, matching the merge's sorted output.
      • Local validation: full catch2 suite; Python suite subsets covering config parsing, schema create, metadata cursors, dump, tiered (strip still active when tiered is enabled), and all test_layered_schema* including the pick-up reproducer — all green.

      When tiered storage is removed

      • This block reduces to the else branch; simplify further to WT_ERR(__wt_metadata_insert(session, uri, fileconf)); and drop filestripped.
      • __wt_config_tiered_strip is deleted along with tiered storage; its remaining callers convert to plain collapse.
      • The metadata byte difference disappears on its own: once the generated bases no longer contain tiered_storage, there is no shared=false left to retain.

      Definition of done

      • Guard merged at the __create_file call site.
      • Instrumented Evergreen run of the many-collections disagg startup workload showing the config-assembly phase statistic reduced by ~35-40% (~5 s off startup pick-up).
      • Full test suite green in CI.

            Assignee:
            Etienne Petrel
            Reporter:
            Etienne Petrel
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: