[KAFKA-229] Monitor the Kafka Source Connector Lag Compared to Source Cluster Created: 02/Jun/21 Updated: 02/Sep/22 Resolved: 10/Aug/22 |
|
| Status: | Closed |
| Project: | Kafka Connector |
| Component/s: | Source |
| Affects Version/s: | 1.5.0 |
| Fix Version/s: | 1.8.0 |
| Type: | Improvement | Priority: | Unknown |
| Reporter: | Diego Rodriguez (Inactive) | Assignee: | Maxim Katcharov |
| Resolution: | Duplicate | Votes: | 7 |
| Labels: | internal-user | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||
| Epic Link: | Monitoring and troubleshooting Kafka Connector | ||||||||||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||||||||||
| Description |
|
Hi Team, At the moment, the Kafka Source Connector provides no easy way to understand how far behind the Connector is in relationship to the associated Change Stream of the Source Cluster. Being aware of that lag and being able to easily implement monitoring and alerting based on it would be really helpful because of the following:
Nowadays, a custom monitoring solution consisting of a consumer that does the following could be implemented as a workaround:
Although feasible, this solution requires development effort and gets more complicated when dealing with a Cluster with more than one Shard. Some solutions that came up after some discussion are:
I'm open to any other suggestions that might make help accomplish this. Regards |
| Comments |
| Comment by Maxim Katcharov [ 10/Aug/22 ] |
|
Closed as duplicate of https://jira.mongodb.org/browse/KAFKA-64 |