Handle parallel marker creation

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Fixed
    • Priority: Major - P3
    • 8.2.0-rc0
    • Affects Version/s: None
    • Component/s: None
    • None
    • Storage Engines, Storage Engines - Server Integration
    • Fully Compatible
    • SESI - 2025-07-08, Addy BBBQ'd Greg - 2025-07-22
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Currently, the initial set of markers is created using an estimation of the maximum size of the oplog. We then add new markers as we write to the oplog. We can maintain this behaviour by creating an initial set of markers asynchronously, and continuing to create new markers as the oplog grows in parallel. Once initial sampling is complete, we can merge the two groups of markers to achieve a full set of markers.

      The current approach to avoid overlap of marker sections is to identify the last entry in the oplog during startup. When sampling (or falling back to collection scanning), use a bounded cursor to create a range from the start to that last entry. This ensures that during initial sampling, no oplog entries beyond the startup timestamp are included. By doing so, we can prevent overlapping markers for any newer oplog entries created after startup.

      This ticket will implement the merge of markers and subsequent testing.

            Assignee:
            Adeline Chen
            Reporter:
            Clarisse Cheah
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: