[KAFKA-316] Kafka Source Connector - Copy Existing Created: 27/May/22  Updated: 27/Oct/23  Resolved: 20/Jul/22

Status: Closed
Project: Kafka Connector
Component/s: Source
Affects Version/s: 1.5.0
Fix Version/s: None

Type: Question Priority: Unknown
Reporter: Rajesh Vinayagam Assignee: Robert Walters
Resolution: Gone away Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

We are trying to leverage Mongo Source Connector for copying the data from a collection.

Below configuration was used and it captures only the changed documents and not the documents from beginning. 

// Configuration
 
{
  "name": "MongoSourceConnectorConnector_0",
  "config": {
    "connector.class": "com.mongodb.kafka.connect.MongoSourceConnector",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "value.converter": "org.apache.kafka.connect.storage.StringConverter",
    "connection.uri": "",
    "database": "demo",
    "collection": "identity",
    "pipeline": "[         {             '$match': {                 'operationType': {'$in': ['insert', 'update', 'replace'], }             }         },         {             '$project': {                 '_id': 1,                 'fullDocument': 1,                 'ns': 1,             }         }     ]",
    "publish.full.document.only": "true",
    "topic.namespace.map": "{\"*\":\"demo.identity\"}",
    "copy.existing": "true"
  }
}

Is there a provision for us to copy the data from the start and do we have any restriction of how far in the past it can support.



 Comments   
Comment by PM Bot [ 20/Jul/22 ]

There hasn't been any recent activity on this ticket, so we're resolving it. Thanks for reaching out! Please feel free to comment on this if you're able to provide more information.

Comment by Robert Walters [ 05/Jul/22 ]

There is no option to specify a point in time you'll always get the entire data in the collection.

Generated at Thu Feb 08 09:06:06 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.