[KAFKA-230] Change stream doesn't recover with errors.tolerance=all on Mongo 4 Created: 02/Jun/21  Updated: 28/Oct/23  Resolved: 09/Jul/21

Status: Closed
Project: Kafka Connector
Component/s: Source
Affects Version/s: 1.5.1, 1.6.0
Fix Version/s: 1.6.0

Type: Bug Priority: Unknown
Reporter: Tim Patterson Assignee: Ross Lawley
Resolution: Fixed Votes: 0
Labels: external-user
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Confluent 5.5.4 (kafka 2.4.x I believe)



 Description   

Currently if we fall off the end of the oplog for whatever reason we get the following error on mongo 4:

INFO An exception occurred when trying to get the next item from the Change Stream: Query failed with error code 280 and error message 'Resume of change stream was not possible, as the resume point may no longer be in the oplog.' on server mongos-profile-1.sailthru.cloud:17020 (com.mongodb.kafka.connect.source.MongoSourceTask)

With errors.tolerance=all the source should auto recover starting with a fresh "change stream".

However this doesn't happen due to:

  1. The resumeTokenNotFound error detection not matching on the returned error.
  2. The error actually being thrown during the reading of the cursor, not the creation of the cursor(where the recovery logic currently lives)

This minimal patch is enough to fix for us https://github.com/tim-patterson/mongo-kafka/commit/a281a6c42256efdea6613e5a2a91bca61cfa75c6

(I'm currently chasing up on my companies open source policies so I'm not sure if I'm going to be in a position to submit a fix/PR)



 Comments   
Comment by Githook User [ 09/Jul/21 ]

Author:

{'name': 'Ross Lawley', 'email': 'ross.lawley@gmail.com', 'username': 'rozza'}

Message: Fixed Source Connector recovery during getMore call (#79)

KAFKA-230 KAFKA-219
Branch: master
https://github.com/mongodb/mongo-kafka/commit/2f1b034437cb140d8d30682169fbb633c98c5623

Comment by Ross Lawley [ 07/Jul/21 ]

PR: https://github.com/mongodb/mongo-kafka/pull/79

Comment by Esha Bhargava [ 04/Jun/21 ]

tpatterson@sailthru.com Thank you for reporting this issue! We'll look into it and get back to you soon.

Generated at Thu Feb 08 09:05:54 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.