Have ASC write the write generation high-water mark so a leader need not scan every table

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Storage Engines - Transactions
    • 30.081
    • None
    • None

      WT-17763 records the high-water mark of write generations a leader has persisted (conn->max_write_gen) as a max_write_gen field in the disaggregated checkpoint metadata, and a node picking a checkpoint up adopts it (metadata.max_write_gen). A node that opens a checkpoint written before the field existed (an upgrade) still has to derive the mark by scanning every file's persisted write generation via __wt_meta_correct_base_write_gen the first time it becomes leader.

      (Note: WT-17763 originally stored the mark in the shared metadata table but moved it into the checkpoint metadata, because writing it into the shared metadata table dirties that table's btree page on every checkpoint that advances the mark and the connection-close checkpoint then discards the page a second time, which the page log rejects as a double free. The checkpoint metadata is a flat descriptor with no btree page, so it has no such lifecycle.)

      ASC (the component that materializes checkpoints) could compute and write the max_write_gen field itself, so the value is always present in a picked-up checkpoint and WiredTiger never has to fall back to the full-table scan at become-leader. Since the field lives in the checkpoint metadata descriptor, ASC can also read it directly without opening a btree, addressing the original request to avoid scanning btrees for it.

      Follow-up to WT-17763. In scope:

      • ASC populates the max_write_gen field in the disaggregated checkpoint metadata at checkpoint materialization.
      • WiredTiger drops the fallback metadata scan at become-leader once the field is guaranteed present.

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Chenhao Qu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: