[SERVER-83185] Close Windows on completely idle streams Created: 30/Oct/23  Updated: 05/Dec/23  Resolved: 15/Nov/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.2.0-rc0

Type: New Feature Priority: Major - P3
Reporter: Joe Niemiec Assignee: Matthew Normyle
Resolution: Fixed Votes: 1
Labels: init-337-m3
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
Assigned Teams:
Atlas Streams
Backwards Compatibility: Fully Compatible
Sprint: Sprint 36
Participants:

 Description   

Today, completely idle streams do not advance the watermark, allowing windows to close. This behavior is preferred in systems that need deterministic results but can produce an edge case with streams that are at times, inactive and will not advance the event-watermarks used to close the windows. In these edge cases, it is possible for an event to be stuck in a window[15min] potentially for many minutes/hours/days before a new event is read from the source, which advances the watermark closing the window and producing the output. 

 

To help solve these edge cases, another option should be configurable, which looks at the wall clock time only when the source inputs are entirely inactive. It should also consider the window time duration as well, as the window should not be closed if its possible events could still be consumed that would advance the watermark time. 

 

ie - 15-minute window, with a sourceIdleness of 30 seconds, should only force the window to close on the windowEndTime + sourceidleness have been reached on the wall. 

 

By default, this configuration would not be used as it creates un-deterministic results.  If the source cannot be connected to for some reason this property should have no effect .



 Comments   
Comment by Matthew Normyle [ 13/Nov/23 ]

kenny.gorman@mongodb.com I'm glad you bring window triggers up.. this feature is really a convenient mixture of Flink's out-of-the-box "event time triggers" and "processing time triggers".

Flink has support for lots of other triggers. We shouldn't try to take them on in this ticket-- but I hope we invest in other types of windows at some point.

Comment by Kenny Gorman [ 01/Nov/23 ]

Should we implement window triggers so we have a generic mechanism?  https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/operators/windows/

Generated at Thu Feb 08 06:51:29 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.