Mongo-Source Connector not able to commit offset sometimes

XMLWordPrintableJSON

    • Type: Question
    • Resolution: Works as Designed
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Hi Team,
      There is a Spring Task application running which responsibility is to copy document from one MongoDB collection to a temp collection. The Mongo-Kafka source connector is configured with temp collection, so whenever data inserted through Spring Task application to the temp collection data will be shipped to Kafka topic through Mongo-Kafka source connector.
      But sometimes data got inserted in the temp collection but not shipped to Kafka topic.
      I can see "new change stream cursor created without offset" in the connect log but sometimes not see
      "Finished commitOffsets successfully in" (when event not posted in Kafka topic)

      Attaching the two connect logs 
      a> offsetCommitted -> when data shipped successfully to Kafka topic
      b> offsetNotCommitted -> when data not shipped to Kafka topic

        1. offsetNotCommitted.PNG
          offsetNotCommitted.PNG
          88 kB
        2. offestCommitted.PNG
          offestCommitted.PNG
          99 kB

            Assignee:
            Ross Lawley
            Reporter:
            Avijit Ghosh
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: