-
Type:
Improvement
-
Resolution: Works as Designed
-
Priority:
Major - P3
-
None
-
Affects Version/s: 1.3.0
-
Component/s: Source
-
None
-
None
-
None
-
None
-
None
-
None
-
None
As per how to recover from an invalid resume token, the only way to recover from an invalid token is to stop the connector, delete the saved offsets, and restart the connector. This will have the connector start a new change stream.
This is a critical weakness, as it causes data loss when an unrecoverable event is encountered - for example, a change stream event that exceeds the default 16MB document size limit.
The source connector already supports the ability to pass a pipeline; could it also be enhanced to allow passing one or more of the options db.watch() supports, such as startAtOperationTime, startAfter, or even resumeAfter? These values could be passed via the json configuration used to create the connector. Supporting startAfter would allow easily skipping a bad record, if the current error message could print out the complete resumeAfter token (I believe it truncates currently).