[SERVER-80349] Support watermark alignment Created: 23/Aug/23  Updated: 29/Oct/23  Resolved: 07/Sep/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Aadesh Patel (Inactive) Assignee: Aadesh Patel (Inactive)
Resolution: Fixed Votes: 0
Labels: init-337-m3
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Assigned Teams:
Atlas Streams
Backwards Compatibility: Fully Compatible
Sprint: Sprint 30, Sprint 31
Participants:

 Description   

Watermark alignment is to prevent a large drift in watermarks across sources. This is only relevant for kafka right now, since we maintain a separate watermark per partition, so this will ensure that the min and max watermarks in the kafka topic partition set never drifts for more than the configured max drift. If a kafka partition is getting consumed significantly faster than another kafka partition, and the watermark drift exceeds the max drift, then all kafka partitions with a watermark that exceeds (min_watermark + max_drift) will be temporarily paused until the partition thats falling behind can catch back up.

This is mostly just relevant to windowed pipelines to prevent the open window state from growing indefinitely because the combined watermark can proceed b/c one partition is falling behind and another partition is way faster.


Generated at Thu Feb 08 06:43:18 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.