-
Type:
Improvement
-
Resolution: Works as Designed
-
Priority:
Unknown
-
None
-
Affects Version/s: None
-
Component/s: Source
-
(copied to CRM)
-
Not Needed
Hi Team!
Although the message for the "Resume Token Not Found" error has been greatly improved thanks to KAFKA-91, it is still not easy to identify which connector throws the exception:
[2021-08-04 10:00:07,176] WARN Failed to resume change stream: Resume of change stream was not possible, as the resume point may no longer be in the oplog. 286 ===================================================================================== If the resume token is no longer available then there is the potential for data loss. Saved resume tokens are managed by Kafka and stored with the offset data. To restart the change stream with no resume token either: * Create a new partition name using the `offset.partition.name` configuration. * Set `errors.tolerance=all` and ignore the erroring resume token. * Manually remove the old offset from its configured storage. Resetting the offset will allow for the connector to be resume from the latest resume token. Using `copy.existing=true` ensures that all data will be outputted by the connector but it will duplicate existing data. ===================================================================================== (com.mongodb.kafka.connect.source.MongoSourceTask:422)
Understanding that one Source Connector opens a single Change Stream to the source MongoDB Cluster, would it be possible to include the Source Connector name within the error message? This would allow for easier discovery of the affected Connector, specially in distributed environments where several other Connectors might be running.
Eventually, logging the associated Source (or Sink) Connector name would be appreciated for any other errors where identifying the associated Connector might be otherwise complicated and painful.
Thanks
Diego