[KAFKA-88] When using pipeline for multiple operationTypes, not all events are published Created: 18/Feb/20  Updated: 27/Oct/23  Resolved: 18/Feb/20

Status: Closed
Project: Kafka Connector
Component/s: None
Affects Version/s: 1.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Jeffrey Sposetti Assignee: Unassigned
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

When using the Kafka Source Connector that watches the deployment but uses a pipeline (with two+ operations), kafka topics are not getting all events.

{
  "name": "testsource",
  "connector.class": "com.mongodb.kafka.connect.MongoSourceConnector",
  "connection.uri": "mongodb://admin:pass@192.168.74.101:27001/?authSource=admin&replicaSet=repl-example&ssl=false",
  "copy.existing": "false",
  "pipeline": "[ { \"$match\": {\"operationType\": \"dropDatabase\", \"operationType\": \"drop\" } } ]"
} 

 

Step to repro

1) Use the source connector configuration above to watch entire deployment and looks for only dropDatabase and drop events.

2) Create database test, create collection test.testone, insert doc into test.testone. No events are added to kafka topics, which is the correct behavior.

3) drop test database. The change stream will produce drop and dropDatabase events but only dropDatabase is published to test kafka topic (and no drop event is published to test.testone kafka topic.

 

EXPECTED BEHAVIOR: test.testone kafka topic should also get a drop event published

 



 Comments   
Comment by Ross Lawley [ 24/Feb/20 ]

Just to note easiest method to filter multiples would be to use a $in query match.

 "pipeline": "[ { \"$match\": {\"operationType\":  { \"$in\": [ \"dropDatabase\",  \"drop\" } } } ]"

Comment by Jeffrey Sposetti [ 18/Feb/20 ]

Works with pipeline:

 
[ { "$match":

{"operationType": RegExp("dropDatabase|drop") }

} ]

Generated at Thu Feb 08 09:05:33 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.