Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-76

Reuse the postBatchResumeToken

XMLWordPrintableJSON

    • Type: Icon: Improvement Improvement
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 1.3.0
    • Affects Version/s: None
    • Component/s: Source
    • None
    • Needed
    • Hide

      Added two new configurations:

      heartbeat.interval.ms=0

      The length of time between sending heartbeat messages to record the post batch resume token when no source records have been published. Improves the resumability of the connector for low volume namespaces. Use 0 to disable.

      heartbeat.topic.name="__mongodb_heartbeats"
      The name of the topic to publish heartbeats to. Defaults to '__mongodb_heartbeats'.

      Note: By default this feature is off, setting the `heartbeat.interval.ms` will turn it on. If there have been no messages in the `heartbeat.interval.ms` time then the post batch resume token is sent to the heartbeat topic. Messages on the heartbeat topic have to be consumed so that the latest offset (post batch resume token) is available.

      Show
      Added two new configurations: heartbeat.interval.ms=0 The length of time between sending heartbeat messages to record the post batch resume token when no source records have been published. Improves the resumability of the connector for low volume namespaces. Use 0 to disable. heartbeat.topic.name="__mongodb_heartbeats" The name of the topic to publish heartbeats to. Defaults to '__mongodb_heartbeats'. Note: By default this feature is off, setting the `heartbeat.interval.ms` will turn it on. If there have been no messages in the `heartbeat.interval.ms` time then the post batch resume token is sent to the heartbeat topic. Messages on the heartbeat topic have to be consumed so that the latest offset (post batch resume token) is available.

      The latest Kafka connector uses Java 3.11 and thus takes advantage of SERVER-35740 (HighWaterMarkToken). Under the hood though, it is the resumeToken (offset) and the corresponding event (matching the change stream filter) that are published as topics. If a connector crash, the offset from the last published Source Record is used as the resumeToken upon restart.

      In certain situation, such as when the connector is listening a dormant db/collection, there is a potential for the resumeToken to be out of the oplog upon restart. Saving the postBatchResumeToken will reduce the likelihood of such a failure to occur

            Assignee:
            ross@mongodb.com Ross Lawley
            Reporter:
            davenson.lombard@mongodb.com Davenson Lombard
            Votes:
            3 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated:
              Resolved: