-
Type:
Task
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: None
-
None
-
Atlas Streams
-
Fully Compatible
-
None
-
0
-
None
-
None
-
None
-
None
-
None
-
None
Add support for a non-resumable single-threaded initialSync phase for one collection.
- Make it possible for a SP to use the specified syntax to describe an initialSync phase with one collection. Gate this feature behind a feature flag.
- The SP should record the current cluster time - T0
- An auxiliary thread should do a bulk copy of the entire collection and decorate the read data as insert change events and pass it on the main SP pipeline. Make it possible to configure this to use / not use snapshot-at-T0 reads.
- After the bulk copy is done, this thread should exit.
- The SP should start its change event processing as normal and read all the change events from T0 onwards.
- Checkpoints should be skipped while the bulk copy is ongoing.
- Add tests to test the e2e flow above
Edit: Removing these from the initial ticket
Checkpoints should be skipped while the bulk copy is ongoing.Track whether initialSync is done or not in the checkpoint state.InitialSync should be skipped after it is done once, if SP is restarted for any reason (edit SP/restart etc)Add support for initialSync stats as described.
- is depended on by
-
SERVER-103318 Investigate different partitioning strategies and select one
-
- In Code Review
-
- is related to
-
SERVER-102307 parallel initialSync assuming id is of type ObjectId
-
- Closed
-