[KAFKA-231] MongoSink Connector is missing messages from Topic Created: 21/Jun/21 Updated: 27/Oct/23 Resolved: 14/Jul/21 |
|
| Status: | Closed |
| Project: | Kafka Connector |
| Component/s: | None |
| Affects Version/s: | 1.5.0 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Unknown |
| Reporter: | Nitin Kapoor | Assignee: | Ross Lawley |
| Resolution: | Gone away | Votes: | 0 |
| Labels: | external-user | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Confluent Kafka |
||
| Attachments: |
|
| Description |
|
Hi , Mongo Sink connector is missing messages (present in mongo changestream and topic in Kafka) from topic . i could see the whole message in the topic as well as in the changestream. What could be the reason.
this token is present, but the sync isnt happening . the sync conf:
I have removed the sensitive information. Its happening for few documents (randomly) not for all . Any lead would be appreciated. Let me know if anything else is needed . BR// |
| Comments |
| Comment by Backlog - Core Eng Program Management Team [ 14/Jul/21 ] | |||||||||
|
There hasn't been any recent activity on this ticket, so we're resolving it. Thanks for reaching out! Please feel free to comment on this if you're able to provide more information. | |||||||||
| Comment by Ross Lawley [ 29/Jun/21 ] | |||||||||
|
Unfortunately the logs didn't provide an insight, as there are no logs from the connector itself. Could you set the error logging in the Kafka connector configuration? The ChangeStreamHandler would have been able to handle the message without issue and convert it into an Update operation. So there is no obvious reason as to why the message on the topic wasn't processed by the connector. Ross | |||||||||
| Comment by Nitin Kapoor [ 24/Jun/21 ] | |||||||||
|
Sure , change stream message which was ignored :
This message is present in the topic and the changestream . I see no errors in the logs on kafka docker compose . MongoDb is present on Atlas ( Mongo cloud , an M10 instance) I have also attached the logs I had . BR// | |||||||||
| Comment by Ross Lawley [ 21/Jun/21 ] | |||||||||
|
Thanks for the ticket - what was the changestream message that wasn't actioned? Kind regards, Ross |