[KAFKA-152] slice and size pipeline commands not working in source connector. Created: 01/Sep/20  Updated: 27/Oct/23  Resolved: 22/Sep/20

Status: Closed
Project: Kafka Connector
Component/s: None
Affects Version/s: 1.2.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Sabari Gandhi Assignee: Ross Lawley
Resolution: Works as Designed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Kafka Connector: 1.2.0
MongoDb version: 3.6.17



 Description   

If I have an array in my document and I wanted to know only the size of the array rather than actual array or to get the top 3 or top 10 rather than full array. I was able to use the pipeline commands slice and size in the mongo cli. But when i use the same in the mongo source connector configuration i don't see any data in the kafka topic. 

 

Interestingly i don't see any error or issue and I also see something like this in the logs which usually confirms data is published to the topic.

Finished commitOffsets successfully in 3 ms.

Can you please confirm what's the valid pipeline configuration to achieve the above scenario. Thanks in advance.



 Comments   
Comment by Ross Lawley [ 15/Sep/20 ]

HI sabari.mgn@gmail.com,

Just to let you know only the following pipeline operatorsare supported by change streams. The source connector uses change streams to provide the CDC events. Withoiut seeing the connector configuration I can't report back on if there is an error or if there is a misconfiguration. In general server errors are reported back to the user.

It may be worth looking at the schema design and how the data is organized when looking for a well architected solution. There are a couple of support avenues open for yourself to get advice going forward:

All the best,

Ross

Comment by Sabari Gandhi [ 01/Sep/20 ]

FYI, I was trying to use the above to overcome the issue reported in https://jira.mongodb.org/browse/KAFKA-151 and was not able to do it.

Generated at Thu Feb 08 09:05:42 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.