copy.existing does not copy all of the existing documents

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Gone away
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Given a database and collection with existing data, when the copy.existing flag is set to true, upon initial configuration of the connector only 10-20% of the documents will be sourced from mongodb into the kafka topic.

      repro:
      1. insert at least 1000 documents into the Stocks.StockData collection
      2. configure the kafka source connector as follows:

      "tasks.max":"1",
      "connector.class":"com.mongodb.kafka.connect.MongoSourceConnector",
      "key.converter":"org.apache.kafka.connect.storage.StringConverter",
      "value.converter":"org.apache.kafka.connect.storage.StringConverter",
      "publish.full.document.only": false,
      "copy.existing" : true,
      "connection.uri":"mongodb://mongo1:27017,mongo2:27017,mongo3:27017",
      "topic.prefix":"stockdata",
      "database":"Stocks",
      "collection":"StockData"

      3. view the messages on the kafka topic, you'll see only about 100 show up even after the connector is left running for a long time

            Assignee:
            Ross Lawley
            Reporter:
            Robert Walters (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: