|
We are trying to leverage Mongo Source Connector for copying the data from a collection.
Below configuration was used and it captures only the changed documents and not the documents from beginning.
// Configuration
|
|
{
|
"name": "MongoSourceConnectorConnector_0",
|
"config": {
|
"connector.class": "com.mongodb.kafka.connect.MongoSourceConnector",
|
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
|
"value.converter": "org.apache.kafka.connect.storage.StringConverter",
|
"connection.uri": "",
|
"database": "demo",
|
"collection": "identity",
|
"pipeline": "[ { '$match': { 'operationType': {'$in': ['insert', 'update', 'replace'], } } }, { '$project': { '_id': 1, 'fullDocument': 1, 'ns': 1, } } ]",
|
"publish.full.document.only": "true",
|
"topic.namespace.map": "{\"*\":\"demo.identity\"}",
|
"copy.existing": "true"
|
}
|
}
|
Is there a provision for us to copy the data from the start and do we have any restriction of how far in the past it can support.
|