Details
-
Bug
-
Resolution: Gone away
-
Major - P3
-
None
-
None
-
None
-
None
Description
Given a database and collection with existing data, when the copy.existing flag is set to true, upon initial configuration of the connector only 10-20% of the documents will be sourced from mongodb into the kafka topic.
repro:
1. insert at least 1000 documents into the Stocks.StockData collection
2. configure the kafka source connector as follows:
"tasks.max":"1",
"connector.class":"com.mongodb.kafka.connect.MongoSourceConnector",
"key.converter":"org.apache.kafka.connect.storage.StringConverter",
"value.converter":"org.apache.kafka.connect.storage.StringConverter",
"publish.full.document.only": false,
"copy.existing" : true,
"connection.uri":"mongodb://mongo1:27017,mongo2:27017,mongo3:27017",
"topic.prefix":"stockdata",
"database":"Stocks",
"collection":"StockData"
3. view the messages on the kafka topic, you'll see only about 100 show up even after the connector is left running for a long time