Use processing time to decide when to take a checkpoint

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Atlas Streams
    • None
    • 3
    • TBD
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Today, we take a checkpoint periodically based on some factors, the main ones being "have we accumulated enough state size" with some additional bounds like atleast once every hour.

      For the non-window case where the state size is typically less, we end up taking a lot of checkpoints, something like once every 5-6 minutes.

      We can will change this to also use processing time (i.e. "cpu time") spent in the pipeline as an additional signal. So the form of the heuristic will be "atleast x secs or atleast y bytes (of state size)" with the additional caps like atleast once per hour and so on.

              Assignee:
              Unassigned
              Reporter:
              Mayuresh Kulkarni
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: