[ds-06.08][Storage Engines (Core)] Change stream support

XMLWordPrintableJSON

    • Storage Engines - Foundations, Storage Engines - Transactions
    • 0
    • StorEng - Defined Pipeline
    • None
    • 4
    • Disag_M13

      For disaggregated storage we will support change streams using the same basic mechansims as attached storage. This means we need to support truncate operations on layered tables. In particular we need:

      • Support for timestamped fast truncates (WT-14778)
      • Layered table support for fast truncate on follower nodes (WT-14679)

      Currently changes streams work by reading information about past operations from the oplog. This is inefficient for a variety of reasons. For Disagg (and possibly 9.0 in general), the server would like to use durable history instead.

      This would require the ability to have different history windows (i.e., oldest timestamps) for different tables. In practice, it may be sufficient to support two different history windows – a default (equivalent to the current 300 second window) and a larger window (24 hours?) for tables that have change streams enabled.

            Assignee:
            Radoslav Kardum
            Reporter:
            Ashu Jaiswal
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: