-
Type:
Task
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: None
-
None
-
Atlas Streams
-
Fully Compatible
-
Sprint 64, Sprint 66
- Update streamMeta with the right timestamp depending on whether the pipeline is using processingTime
- Update WindowAwareOperator::assignWindowsAndProcessDataMsg methods to ensure that in processingTime window case, late/dlq message logic is skipped.
Make sure that the stats response has the correct watermark for the source , and in the kafka case, also for each partition. No additional change should be needed for this. - Update WindowAwareOperator::processWatermarkMsg so that windows are closed based on the right timestamp from the watermark msg
- Add checkpoint support. The only change should be around renaming the WatermarkState::eventTimeMs to be WatermarkState::watermarkTimestampMs. Do this using the cpp_name IDL feature to avoid backwards compatibility issues.
- Add jstest for a pipeline with change stream source and processing time windows and check that docs are being assigned to windows correctly
- Add jstest to check that windows are closing as expected
- Add jstests to checkpoint-in-the middle suite to test checkpointing of processingTime windows
- Add jstests to check that windows close regardless of idle timeouts, and there are no DLQ’d messages due to messages being “late.
- Add jstests to validate that the stats reply has the processingTime watermark
- has to be done after
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
- has to be done before
-
-
- Investigating
-
-
-
- Needs Scheduling
-
- is related to
-
-
- Closed
-
- mentioned in
-
Page Loading...