[KAFKA-341] Pipeline doesn't work in mongo source connector when Publish full document only: true Created: 15/Nov/22  Updated: 17/Nov/22  Resolved: 16/Nov/22

Status: Closed
Project: Kafka Connector
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Question Priority: Minor - P4
Reporter: Jayanth Culli Assignee: Ross Lawley
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File image-2022-11-16-02-26-10-345.png    
Issue Links:
Related

 Description   

I have set publish full document only to true. When I try to exclude fields from the full document using the pipeline, it doesn't work. Is there a workaround for excluding fields when publishing full document?



 Comments   
Comment by Jayanth Culli [ 17/Nov/22 ]

When we set publish full document only: true, the pipeline suggested by you doesn't work. Verified it myself.

 

 

Comment by Ross Lawley [ 16/Nov/22 ]

Hi jayanth.culli@sliceit.com,

Thank you for reaching out. The pipeline is applied to the change stream event and not just the fullDocument. So you will have to update the configuration with that in mind eg:

pipeline=[{"$project": { "fullDocument.phone": 0 } }]

I've opened a docs ticket to clarify what publish.full.document.only does and its only a helper that returns just the fullDocument field from the change stream event.

For future reference I wanted to give you some better resources to get questions answered:

  • Our MongoDB support portal, located at support.mongodb.com
  • Our MongoDB community portal, located here
  • If you are an Atlas customer, there is free support offered 24/7 in the lower right hand corner of the UI

All the best,

Ross Lawley

Generated at Thu Feb 08 09:06:09 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.