[KAFKA-95] "Unrecognized field: startAfter" with Source connector Created: 06/Apr/20  Updated: 28/Oct/23  Resolved: 08/Apr/20

Status: Closed
Project: Kafka Connector
Component/s: None
Affects Version/s: 1.0.1
Fix Version/s: 1.1

Type: Bug Priority: Major - P3
Reporter: Abhishek Gupta Assignee: Ross Lawley
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

MongoDB 3.6
Kafka 2.3.0



 Description   

I am trying to configure a Source connector. Here are the steps

  1. configure and create source connector - worked fine
  2. create a document in mongodb - worked fine. Confirmed that the document was pushed in the target Kafka topic
  3. restart Kafka connect - I see this error

 

INFO Resuming the change stream after the previous offset (com.mongodb.kafka.connect.source.MongoSourceTask:234)
INFO Failed to resume change stream: Unrecognized field: startAfter 9 (com.mongodb.kafka.connect.source.MongoSourceTask:253)

I am using MongoDB 3.6 and I see that startAfter is supported for 4.2 and above (https://docs.mongodb.com/manual/changeStreams/#startafter-for-change-streams). I checked the code[https://github.com/mongodb/mongo-kafka/blob/d89f0c0ba0b8a365fbcc0536dfa93d1728e4d933/src/main/java/com/mongodb/kafka/connect/source/MongoSourceTask.java#L253] but it's difficult to figure out since all the exception handling is based on error codes. 

Not sure why startAfter is coming into picture in this case



 Comments   
Comment by Abhishek Gupta [ 12/Apr/20 ]

Thank you! Good to see its included in 1.1 release. Out of curiosity, when is the 1.1 release going out? 

Comment by Ross Lawley [ 08/Apr/20 ]

I've added an extra check for errorCode 9 - Unrecognized field. This should ensure that resumeAfter is used instead.

Comment by Githook User [ 08/Apr/20 ]

Author:

{'name': 'Ross Lawley', 'email': 'ross.lawley@gmail.com', 'username': 'rozza'}

Message: Updated startAfter support check

KAFKA-95
Branch: master
https://github.com/mongodb/mongo-kafka/commit/3e324722f403e4f4106ca13ae8019fe59c7f6b1d

Comment by Ross Lawley [ 07/Apr/20 ]

Hi abhirockzz@gmail.com,

I've tested on both 3.6.10 and 3.6.17 and I don't the "Unrecognized field: startAfter 9 " error. Please could you confirm the exact version of MongoDB you are using?

Kind Regards,

Ross

Comment by Ross Lawley [ 07/Apr/20 ]

Hi,

Apologies, that is definitely a bug with the handling of resuming a changestream. Restarting the connector will attempt to restart from the last seen offset. However, MongoDB 3.6 handles the startAfter field differently than expected - a fix will be issued in the next release.

Ross

Generated at Thu Feb 08 09:05:34 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.