[KAFKA-96] Source Connector: The resume token UUID does not exist Created: 09/Apr/20  Updated: 11/Sep/20  Resolved: 11/Sep/20

Status: Closed
Project: Kafka Connector
Component/s: Source
Affects Version/s: None
Fix Version/s: 1.3.0

Type: Bug Priority: Major - P3
Reporter: Pietro Partescano Assignee: Ross Lawley
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Component: Confluent Kafka Connector
Version: 1.0.1

MongoDB version: 3.6.16, sharded cluster 


Issue Links:
Duplicate
duplicates KAFKA-76 Reuse the postBatchResumeToken Closed
duplicates KAFKA-105 Support errors.tolerance Closed
Epic Link: Error Handling

 Description   

Component: Confluent Kafka Connector
Version: 1.0.1

MongoDB version: 3.6.16, sharded cluster 

Related tickets: 

https://jira.mongodb.org/browse/SERVER-32088

https://jira.mongodb.org/browse/SERVER-32029

Issue: 

Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:179)

com.mongodb.MongoCommandException: Command failed with error 40615 (Location40615): 'The resume token UUID does not exist. Has the collection been dropped?' on server xxx. The full response is {"ok": 0.0, "errmsg": "The resume token UUID does not exist. Has the collection been dropped?", "code": 40615, "codeName": "Location40615", "operationTime": {"$timestamp": {"t": 1584614769, "i": 95, "$clusterTime": {"clusterTime": {"$timestamp": {"t": 1584614769, "i": 95}}, "signature": {"hash":

{"$binary": "2BrRm3m276aXzvoFKW+R/TnxNMU=", "$type": "00"}

, "keyId": {"$numberLong": "6763316478826514148"}}}}}}

 

The error happens when a changeStream generated by the Kafka Connector cannot find the collection on the target server via the mongos. This is a known bug which is fixed in MongoDB 4+ as seen in ticket SERVER-32029 and a backport for this will not be implemented. Upgrading to MongoDB 4+ is not an option for the time being due to time constraints

 

Questions:

We were wondering if there was a workaround for this specific issue? We use tag based sharding and we shard the collections based on this tag. We have one active shard currently ->  we know which shard is receiving writes and where the data is located. We thought it might be possible to tail the oplog of the shard primary in the active shard using the Kafka Connector, receive the data and dump it to Kafka, thoughts on this / is this even plausible? Is the only way to receive data through mongos using changeStreams initiated by the Kafka Connector when we’re dealing with a sharded MongoDB cluster? 

 

Workflow:

MongoDB Shard/ReplicaSet Oplog (current active shard) → KafkaConnector →Kafka Cluster

 



 Comments   
Comment by Ross Lawley [ 11/Sep/20 ]

Better support for errors.tolerance in the source connector (KAFKA-105) and / or use of the postBatchResumeToken (KAFKA-76) will help users manage when resume tokens are missing.

Generated at Thu Feb 08 09:05:34 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.