Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-179

copy.existing does not copy all of the existing documents

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Gone away
    • Icon: Major - P3 Major - P3
    • None
    • None
    • None
    • None

    Description

      Given a database and collection with existing data, when the copy.existing flag is set to true, upon initial configuration of the connector only 10-20% of the documents will be sourced from mongodb into the kafka topic.

      repro:
      1. insert at least 1000 documents into the Stocks.StockData collection
      2. configure the kafka source connector as follows:

      "tasks.max":"1",
      "connector.class":"com.mongodb.kafka.connect.MongoSourceConnector",
      "key.converter":"org.apache.kafka.connect.storage.StringConverter",
      "value.converter":"org.apache.kafka.connect.storage.StringConverter",
      "publish.full.document.only": false,
      "copy.existing" : true,
      "connection.uri":"mongodb://mongo1:27017,mongo2:27017,mongo3:27017",
      "topic.prefix":"stockdata",
      "database":"Stocks",
      "collection":"StockData"

      3. view the messages on the kafka topic, you'll see only about 100 show up even after the connector is left running for a long time

      Attachments

        Activity

          People

            ross@mongodb.com Ross Lawley
            robert.walters@mongodb.com Robert Walters
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: