[KAFKA-141] Mongo Source Connector seeks non-existing topic Created: 11/Aug/20 Updated: 27/Oct/23 Resolved: 03/Sep/20 |
|
| Status: | Closed |
| Project: | Kafka Connector |
| Component/s: | Source |
| Affects Version/s: | 1.2.0 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | J vH | Assignee: | Ross Lawley |
| Resolution: | Gone away | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Description |
|
A general outline of this issue has been posted on stackoverflow. In a nutshell, the Kafka producer that underlies the MongoDB Source Connector sometimes points to the wrong (and thus non-existing) topic. The database name - that is by definition part of the topic name - is prefixed by an inexplicable number. This happens during the deployment of the Mongo Source Connector, but it also popped-up during the restart of one of our Kafka Connect nodes. Please find the logs (nothing is logged on DEBUG level) and our connector configuration attached. Note that by changing the name of the connector the bug eventually does not occur. This flaky behavior is a blocker for us to go to production. |
| Comments |
| Comment by Ross Lawley [ 11/Aug/20 ] |
|
Hi jeffrey.vanhelden@thewarehouse.co.nz, Could it be to do with the partition metadata being stored via zookeeper, that has not been replicated to all nodes? There is nothing that I can see in the source connector that would produce a topic name as such. You may need to check the Kafka logs. Ross |