Change stream response documents must adhere to the 16MB BSON : https://docs.mongodb.com/manual/administration/change-streams-production-recommendations/

Currently Kafka connector:

1) Doesn't create a error or exception but has an info message:

 INFO An exception occurred when trying to get the next item from the changestream. (com.mongodb.kafka.connect.source.MongoSourceTask) INFO An exception occurred when trying to get the next item from the changestream. (com.mongodb.kafka.connect.source.MongoSourceTask)kfc-mongodb-7bbc79cb64-hdddg kfc-mongodb 2020-07-09T16:29:47.376320977Z com.mongodb.MongoCommandException: Command failed with error 10334 (Location10334): 'BSONObj size: 19376544 (0x127A9A0) is invalid. Size must be between 0 and 16793600(16MB) First element: _id: { _data: BinData(0, 825F0745F700000005461E5F69640031035342000004BC005A100458C65EB8491243558BB64BDAC2914E5204), _typeBits: BinData(0, 02) }' on server db-core-1.ebs.us-east-1.stage-us.int.evbg.io:29001. The full response is {"operationTime": {"$timestamp": {"t": 1594312187, "i": 87, "ok": 0.0, "errmsg": "BSONObj size: 19376544 (0x127A9A0) is invalid. Size must be between 0 and 16793600(16MB) First element: _id: { _data: BinData(0, 825F0745F700000005461E5F69640031035342000004BC005A100458C65EB8491243558BB64BDAC2914E5204), _typeBits: BinData(0, 02) }", "code": 10334, "codeName": "Location10334", "$clusterTime": {"clusterTime": {"$timestamp": {"t": 1594312187, "i": 87}}, "signature": {"hash":

{"$binary": "WPjr3+K8zh6NDOpkTmK6rNvmM4M=", "$type": "00"}

, "keyId": {"$numberLong": "6828579125364523010"}}}}}}

2. Task status is still running but it fails to process change stream

kfc-mongodb-7bbc79cb64-hdddg kfc-mongodb 2020-07-09T15:59:57.191098599Z [2020-07-09 15:59:57,190] INFO Failed to resume change stream: BSONObj size: 19224166 (0x1255666) is invalid. Size must be between 0 and 16793600(16MB) First element: _id: { _data: BinData(0, 825F073D080000000E461E5F69640031035342000004BC005A100458C65EB8491243558BB64BDAC2914E5204), _typeBits: BinData(0, 02) } 10334 (com.mongodb.kafka.connect.source.MongoSourceTask)

 

Regarding error handling, this seems related to https://jira.mongodb.org/browse/KAFKA-89

Is there a way to have connector resilient of this issues with improved error handling and task status. Thanks in advance.