Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-282

Mongo-Source Connector not able to commit offset sometimes

    XMLWordPrintableJSON

Details

    • Icon: Question Question
    • Resolution: Works as Designed
    • Icon: Major - P3 Major - P3
    • None
    • None
    • None

    Description

      Hi Team,
      There is a Spring Task application running which responsibility is to copy document from one MongoDB collection to a temp collection. The Mongo-Kafka source connector is configured with temp collection, so whenever data inserted through Spring Task application to the temp collection data will be shipped to Kafka topic through Mongo-Kafka source connector.
      But sometimes data got inserted in the temp collection but not shipped to Kafka topic.
      I can see "new change stream cursor created without offset" in the connect log but sometimes not see
      "Finished commitOffsets successfully in" (when event not posted in Kafka topic)

      Attaching the two connect logs 
      a> offsetCommitted -> when data shipped successfully to Kafka topic
      b> offsetNotCommitted -> when data not shipped to Kafka topic

      Attachments

        1. offestCommitted.PNG
          offestCommitted.PNG
          99 kB
        2. offsetNotCommitted.PNG
          offsetNotCommitted.PNG
          88 kB

        Activity

          People

            ross@mongodb.com Ross Lawley
            avi54work@gmail.com Avijit Ghosh
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: