[KAFKA-302] Extended debugging for Kafka Connector Created: 17/Mar/22  Updated: 28/Oct/23  Resolved: 18/Aug/22

Status: Closed
Project: Kafka Connector
Component/s: Sink, Source
Affects Version/s: None
Fix Version/s: 1.8.0

Type: Epic Priority: Major - P3
Reporter: Robert Walters Assignee: Maxim Katcharov
Resolution: Fixed Votes: 0
Labels: internal-user
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Quarter: FY23Q3
Start date:
End date:
Calendar Time: 2 weeks, 4 days
Scope Cost Estimate: 3
Cost to Date: 1
Final Cost Estimate: 3
Cost Threshold %: 100
Detailed Project Statuses:

Engineer(s): Maxim

Summary: Improve logging for the Kafka connector

2022-08-09: 

Status update:

  • First set of logging improvements are in review
  • Add connector id to all log messages is up next

Rationale for delays:

  • No delays

Risks:

  • No risks


 Description   

Information To Be Logged

  • For Both Source and Sink
    • Log the timestamp and execution time (when applicable) for all the events written under this extended debugging feature.
    • Any event that is currently logged as part of the standard Kafka Connect logging should also be written to the target MongoDB deployment defined by this extended debugging feature.
    • Log the key and value of the related Kafka message.
  • For the Sink Connector
    • Log which post-processors were executed, and in which order.
    • Log the execution time of the associated write operation in MongoDB.
  • For the Source Connector
    • Log the most recent Resume Token and/or the Heartbeat Token value.
    • Log the execution time of the `getMore` requests on the Change Stream cursor.


 Comments   
Comment by Maxim Katcharov [ 03/Aug/22 ]

Of the information in the current description, the following are now logged as part of https://github.com/mongodb/mongo-kafka/pull/112

  • For the Sink Connector
    • Log the execution time of the associated write operation in MongoDB.
  • For the Source Connector
    • Log the most recent Resume Token and/or the Heartbeat Token value.
    • Log the execution time of the `getMore` requests on the Change Stream cursor.
Generated at Thu Feb 08 09:06:04 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.