Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-88

When using pipeline for multiple operationTypes, not all events are published

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Works as Designed
    • Icon: Major - P3 Major - P3
    • None
    • 1.0
    • None
    • None

    Description

      When using the Kafka Source Connector that watches the deployment but uses a pipeline (with two+ operations), kafka topics are not getting all events.

      {
        "name": "testsource",
        "connector.class": "com.mongodb.kafka.connect.MongoSourceConnector",
        "connection.uri": "mongodb://admin:pass@192.168.74.101:27001/?authSource=admin&replicaSet=repl-example&ssl=false",
        "copy.existing": "false",
        "pipeline": "[ { \"$match\": {\"operationType\": \"dropDatabase\", \"operationType\": \"drop\" } } ]"
      } 

       

      Step to repro

      1) Use the source connector configuration above to watch entire deployment and looks for only dropDatabase and drop events.

      2) Create database test, create collection test.testone, insert doc into test.testone. No events are added to kafka topics, which is the correct behavior.

      3) drop test database. The change stream will produce drop and dropDatabase events but only dropDatabase is published to test kafka topic (and no drop event is published to test.testone kafka topic.

       

      EXPECTED BEHAVIOR: test.testone kafka topic should also get a drop event published

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            jeffrey.sposetti@mongodb.com Jeffrey Sposetti
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: