[Streams] Support large single group accumulator checkpoint

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Atlas Streams
    • None
    • 3
    • None
    • None
    • None
    • None
    • None
    • None

      The existing checkpoint code doesn't accommodate single large accumulator state larger than 128MB. In Atlas stream processing, the maximum size for a single large accumulator stat is 1GB. We need to add support for checkpointing these large states. 

      https://github.com/10gen/mongo/blob/a01f7c5fd62674c1bbc5531ae35f23b007690cad/src/mongo/db/modules/enterprise/src/streams/exec/checkpoint/local_disk_checkpoint_storage.cpp#L118

      https://github.com/10gen/mongo/blob/a01f7c5fd62674c1bbc5531ae35f23b007690cad/src/mongo/db/modules/enterprise/src/streams/exec/checkpoint/local_disk_checkpoint_storage.h#L32

            Assignee:
            Mayuresh Kulkarni
            Reporter:
            Harendra Chawla
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: