[KAFKA-152] slice and size pipeline commands not working in source connector. Created: 01/Sep/20 Updated: 27/Oct/23 Resolved: 22/Sep/20 |
|
| Status: | Closed |
| Project: | Kafka Connector |
| Component/s: | None |
| Affects Version/s: | 1.2.0 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Sabari Gandhi | Assignee: | Ross Lawley |
| Resolution: | Works as Designed | Votes: | 1 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Kafka Connector: 1.2.0 |
||
| Description |
|
If I have an array in my document and I wanted to know only the size of the array rather than actual array or to get the top 3 or top 10 rather than full array. I was able to use the pipeline commands slice and size in the mongo cli. But when i use the same in the mongo source connector configuration i don't see any data in the kafka topic.
Interestingly i don't see any error or issue and I also see something like this in the logs which usually confirms data is published to the topic. Finished commitOffsets successfully in 3 ms. Can you please confirm what's the valid pipeline configuration to achieve the above scenario. Thanks in advance. |
| Comments |
| Comment by Ross Lawley [ 15/Sep/20 ] |
|
Just to let you know only the following pipeline operatorsare supported by change streams. The source connector uses change streams to provide the CDC events. Withoiut seeing the connector configuration I can't report back on if there is an error or if there is a misconfiguration. In general server errors are reported back to the user. It may be worth looking at the schema design and how the data is organized when looking for a well architected solution. There are a couple of support avenues open for yourself to get advice going forward:
All the best, Ross |
| Comment by Sabari Gandhi [ 01/Sep/20 ] |
|
FYI, I was trying to use the above to overcome the issue reported in https://jira.mongodb.org/browse/KAFKA-151 and was not able to do it. |