Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-94705

Processing time windows

    • Atlas Streams
    • Sprint 58

      Multiple customers have pointed to a desire to have a process time window that is not driven by watermarks but rather by wall clock time. Both American Airlines and Figured have this desire, today they have been using idleTimeout in a hacky way to get behavior that is close to this, but the problem with using idleTimeout is it is still a watermark driven window and will take different behavoir if the watermark is past for an event forcing a DLQ of the event. This behavior isn't desired as they simply wish to process all messages that show up regardless of event time and regardless of order. Events should never go to a DLQ for being late. 

      The processWindow takes only a interval, pipeline, and the timeMode. The interval is the number of seconds that the window remains open, tracked by the wall clock time of the server. Tumbling and Hopping windows can have this behavior enabled by using the timeMode property. This would then allow the window to function using process time ignoring the event watermark and simply tracking to the wallclock. The default for timeMode is 'event'

      $tumblingWindow: {    
      interval: {...},   
      timeMode: "event|processing",
       pipeline : [...] }

      When using processing timeMode the windows are aligned to the top of the hour just like the current tumbling/hopping window behavior, it will NOT start when the first event arrives, rather it is going forever and starts when the processor is running independent to if events are coming in. So if you have the following tumbling window with a 5 minute interval the first window will be from 10:00-10:05, and after 5 minutes of server time will close and open another window from 10:05-10:10 and so on. 

      $tumblingWindow:{          
             interval: {size: NumberInt(5), unit: "minute"},   
            timeMode: "processing",
             pipeline : [...]
      }
      

       

            Assignee:
            sandeep.dhoot@mongodb.com Sandeep Dhoot
            Reporter:
            joe.niemiec@mongodb.com Joe Niemiec
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: