Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-102099

Initial plumbing for InitialSync

    • Type: Icon: Task Task
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 8.2.0-rc0
    • Affects Version/s: None
    • Component/s: None
    • None
    • Atlas Streams
    • Fully Compatible
    • None
    • 0
    • None
    • None
    • None
    • None
    • None
    • None

      Add support for a non-resumable single-threaded initialSync phase for one collection.

      • Make it possible for a SP to use the specified syntax to describe an initialSync phase with one collection. Gate this feature behind a feature flag.
      • The SP should record the current cluster time - T0
      • An auxiliary thread should do a bulk copy of the entire collection and decorate the read data as insert change events and pass it on the main SP pipeline. Make it possible to configure this to use / not use snapshot-at-T0 reads.
      • After the bulk copy is done, this thread should exit.
      • The SP should start its change event processing as normal and read all the change events from T0 onwards.
      • Checkpoints should be skipped while the bulk copy is ongoing.
      • Add tests to test the e2e flow above

      Edit: Removing these from the initial ticket

      • Checkpoints should be skipped while the bulk copy is ongoing.
      • Track whether initialSync is done or not in the checkpoint state.
      • InitialSync should be skipped after it is done once, if SP is restarted for any reason (edit SP/restart etc)
      • Add support for initialSync stats as described

            Assignee:
            nathan.frank@mongodb.com Nathan Frank
            Reporter:
            mayuresh.kulkarni@mongodb.com Mayuresh Kulkarni
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: