Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-182

Source Connector ability to skip bad resumeAfter token

    • Type: Icon: Improvement Improvement
    • Resolution: Works as Designed
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 1.3.0
    • Component/s: Source
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      As per how to recover from an invalid resume token, the only way to recover from an invalid token is to stop the connector, delete the saved offsets, and restart the connector. This will have the connector start a new change stream.

      This is a critical weakness, as it causes data loss when an unrecoverable event is encountered - for example, a change stream event that exceeds the default 16MB document size limit. 

      The source connector already supports the ability to pass a pipeline; could it also be enhanced to allow passing one or more of the options db.watch() supports, such as startAtOperationTime, startAfter, or even resumeAfter? These values could be passed via the json configuration used to create the connector. Supporting startAfter would allow easily skipping a bad record, if the current error message could print out the complete resumeAfter token (I believe it truncates currently). 

            Assignee:
            ross@mongodb.com Ross Lawley
            Reporter:
            bmakana@indeed.com Branden Makana
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: