[KAFKA-231] MongoSink Connector is missing messages from Topic Created: 21/Jun/21  Updated: 27/Oct/23  Resolved: 14/Jul/21

Status: Closed
Project: Kafka Connector
Component/s: None
Affects Version/s: 1.5.0
Fix Version/s: None

Type: Bug Priority: Unknown
Reporter: Nitin Kapoor Assignee: Ross Lawley
Resolution: Gone away Votes: 0
Labels: external-user
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Confluent Kafka


Attachments: File compose_filter.7z    

 Description   

Hi ,

Mongo Sink connector is missing messages (present in mongo changestream and topic in Kafka) from topic .

i could see the whole message in the topic as well as in the changestream.

What could be the reason.

{"_data": "8260CC87A9000000092B022C0100296E5A1004A1F0D3C981B54CD59F248E484B6EEC3746645F6964006460CC4F9343617B00086257740004"},

this token is present, but the sync isnt happening .

the sync conf:

{
  "name": "prod-organizationmanagementSink",
  "connector.class": "com.mongodb.kafka.connect.MongoSinkConnector",
  "tasks.max": "1",
  "topics.regex": "^prod.REMOVED.*",
  "connection.uri": "REMOVED",
  "database": "REMOVED,
  "change.data.capture.handler": "com.mongodb.kafka.connect.sink.cdc.mongodb.ChangeStreamHandler"
}

I have removed the sensitive information.

Its happening for few documents (randomly) not for all .

Any lead would be appreciated. Let me know if anything else is needed .

BR//
Nitin



 Comments   
Comment by Backlog - Core Eng Program Management Team [ 14/Jul/21 ]

There hasn't been any recent activity on this ticket, so we're resolving it. Thanks for reaching out! Please feel free to comment on this if you're able to provide more information.

Comment by Ross Lawley [ 29/Jun/21 ]

Hi nitin.kapoor@inclusio.io,

Unfortunately the logs didn't provide an insight, as there are no logs from the connector itself.

Could you set the error logging in the Kafka connector configuration?
Also setting the logging level for Kafka connect may provide more information.

The ChangeStreamHandler would have been able to handle the message without issue and convert it into an Update operation. So there is no obvious reason as to why the message on the topic wasn't processed by the connector.

Ross

Comment by Nitin Kapoor [ 24/Jun/21 ]

Sure , 

change stream message which was ignored :

compose_filter.7z

{"txnNumber": {"$numberLong": "1"}, 
 "lsid": {"id": {"$binary": "kzv5rbTUTkqPqe5GOcETeA==", "$type": "04"}, "uid": {"$binary": "bbF9aT/Zk+inYXjkL9KKRseG4Jr0PjrL4jtc5IGFD7k=", "$type": "00"}}, 
  "_id": {"_data": "8260CA4A4B000000012B022C01002C07B46E5A1004B20BE4717C774434AEB9E8D3BC83F08C46645F6964006460CA4A46A7C08C0009FB6C1C0004"}, 
   "operationType": "update", 
   "clusterTime": {"$timestamp": {"t": 1623870027, "i": 1}}, 
   "ns": {"db": "organizationmanagement", "coll": "organizationindividualgroups"}, 
   "documentKey": {"_id": {"$oid": "60ca4a46a7c08c0009fb6c1c"}}, 
   "updateDescription": {"updatedFields": {"manager": {"$oid": "60ca3e5fdc5c2c0009862b5c"}}, 
   "removedFields": []}}

This message is present in the topic and the changestream .

I see no errors in the logs on kafka docker compose .

MongoDb is present on Atlas ( Mongo cloud , an M10 instance)

I have also attached the logs I had .

BR//
Nitin

Comment by Ross Lawley [ 21/Jun/21 ]

HI nitin.kapoor@inclusio.io,

Thanks for the ticket - what was the changestream message that wasn't actioned?
Do you have any logs for the period of time? From Kafka, Kafka Connect or MongoDB?

Kind regards,

Ross

Generated at Thu Feb 08 09:05:54 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.