Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-88

When using pipeline for multiple operationTypes, not all events are published

    • Type: Icon: Bug Bug
    • Resolution: Works as Designed
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 1.0
    • Component/s: None
    • Labels:
      None

      When using the Kafka Source Connector that watches the deployment but uses a pipeline (with two+ operations), kafka topics are not getting all events.

      {
        "name": "testsource",
        "connector.class": "com.mongodb.kafka.connect.MongoSourceConnector",
        "connection.uri": "mongodb://admin:pass@192.168.74.101:27001/?authSource=admin&replicaSet=repl-example&ssl=false",
        "copy.existing": "false",
        "pipeline": "[ { \"$match\": {\"operationType\": \"dropDatabase\", \"operationType\": \"drop\" } } ]"
      } 

       

      Step to repro

      1) Use the source connector configuration above to watch entire deployment and looks for only dropDatabase and drop events.

      2) Create database test, create collection test.testone, insert doc into test.testone. No events are added to kafka topics, which is the correct behavior.

      3) drop test database. The change stream will produce drop and dropDatabase events but only dropDatabase is published to test kafka topic (and no drop event is published to test.testone kafka topic.

       

      EXPECTED BEHAVIOR: test.testone kafka topic should also get a drop event published

       

            Assignee:
            Unassigned Unassigned
            Reporter:
            jeffrey.sposetti@mongodb.com Jeffrey Sposetti
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: