Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-89145

Always update Kafka consumer group offsets during stop

    • Type: Icon: Task Task
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 8.1.0-rc0
    • Affects Version/s: None
    • Component/s: None
    • Atlas Streams
    • Fully Compatible
    • Sprint 52

      A follow up to: https://github.com/10gen/mongo/pull/20756

      1. In order to commit the Kafka consumer group offsets one last time, during stopStreamProcessor, mongostream should wait for a notifyFlush call from the Agent. An alternative simpler implementation is just always commit the consumer group offsets after locally writing the final checkpoint, don't wait for notifyFlush from Agent. This has some risk of updating the consumer group offsets before the checkpoint upload to S3 succeeds.

      Also, make a follow up ticket for:

      1. For the very first checkpoint, mongostream should wait for it to be flushed before beginning output.

            Assignee:
            matthew.normyle@mongodb.com Matthew Normyle
            Reporter:
            matthew.normyle@mongodb.com Matthew Normyle
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: