[KAFKA-141] Mongo Source Connector seeks non-existing topic Created: 11/Aug/20  Updated: 27/Oct/23  Resolved: 03/Sep/20

Status: Closed
Project: Kafka Connector
Component/s: Source
Affects Version/s: 1.2.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: J vH Assignee: Ross Lawley
Resolution: Gone away Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

A general outline of this issue has been posted on stackoverflow.
https://stackoverflow.com/questions/62441841/topic-mapping-error-kafka-connect-mongo-source-connector

In a nutshell, the Kafka producer that underlies the MongoDB Source Connector sometimes points to the wrong (and thus non-existing) topic. The database name - that is by definition part of the topic name - is prefixed by an inexplicable number. 

This happens during the deployment of the Mongo Source Connector, but it also popped-up during the restart of one of our Kafka Connect nodes.

Please find the logs (nothing is logged on DEBUG level) and our connector configuration attached.

Note that by changing the name of the connector the bug eventually does not occur. This flaky behavior is a blocker for us to go to production.



 Comments   
Comment by Ross Lawley [ 11/Aug/20 ]

Hi jeffrey.vanhelden@thewarehouse.co.nz,

Could it be to do with the partition metadata being stored via zookeeper, that has not been replicated to all nodes? There is nothing that I can see in the source connector that would produce a topic name as such. You may need to check the Kafka logs.

Ross

Generated at Thu Feb 08 09:05:40 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.