MongoSourceConnector v1.12.0 does not produce tombstone messages

XMLWordPrintableJSON

    • Type: Question
    • Resolution: Gone away
    • Priority: Unknown
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • None
    • Java Drivers
    • None
    • None
    • None
    • None
    • None
    • None

      I am trying out MongoSourceConnector using the tutorial at https://www.mongodb.com/docs/kafka-connector/current/tutorials/source-connector/

      I tried a source configuration as below:

      ```
      {
        "name": "mysource",
        "config":

      {    "connector.class": "com.mongodb.kafka.connect.MongoSourceConnector",     "connection.uri": "mongodb://mongo1",     "topic.namespace.map": "\{\"*\":\"changelog.mongo.kafka.connect\"}

      ",
          "publish.full.document.only": true,
          "publish.full.document.only.tombstones.on.delete": true,
          "change.stream.document.key.as.key": true
        }
      }
      ```

      I then created and updated and deleted documents in the collection Tutorial1.orders.

      Using kafkacat I can see messages being published to the topic "changelog.mongo.kafka.connect". However I don't see any message when I delete documents from the collection.

      ```

      $ kafkacat b localhost:9092 -f '\nPartition: %p\tOffset: %o\n\nKey (%K bytes):\n\n%k\t\n\nValue (%S bytes):\n\n%s\n------------------------\n\n' -t changelog.mongo.kafka.connect 
      % Auto-selecting Consumer mode (use -P or -C to override)

      Partition: 0    Offset: 0

      Key (109 bytes):

      {"schema":\{"type":"string","optional":false}

      ,"payload":"{\"_id\": {\"$oid\": \"6656b7f7c0831732ce3027d2\"}}"}    

      Value (148 bytes):

      {"schema":\{"type":"string","optional":false}

      ,"payload":"{\"_id\":

      {\"$oid\": \"6656b7f7c0831732ce3027d2\"}

      , \"order_id\": 1, \"item\": \"coffee\"}"}
      -------------------------

      Partition: 0    Offset: 1

      Key (109 bytes):

      {"schema":\{"type":"string","optional":false}

      ,"payload":"{\"_id\": {\"$oid\": \"6656b7f7c0831732ce3027d2\"}}"}    

      Value (145 bytes):

      {"schema":\{"type":"string","optional":false}

      ,"payload":"{\"_id\":

      {\"$oid\": \"6656b7f7c0831732ce3027d2\"}

      , \"order_id\": 1, \"item\": \"tea\"}"}
      -------------------------

      Partition: 0    Offset: 2

      Key (109 bytes):

      {"schema":\{"type":"string","optional":false}

      ,"payload":"{\"_id\": {\"$oid\": \"6656b8fac0831732ce3027d4\"}}"}    

      Value (150 bytes):

      {"schema":\{"type":"string","optional":false}

      ,"payload":"{\"_id\":

      {\"$oid\": \"6656b8fac0831732ce3027d4\"}

      , \"order_id\": 100, \"item\": \"coffee\"}"}
      -------------------------

      Partition: 0    Offset: 3

      Key (109 bytes):

      {"schema":\{"type":"string","optional":false}

      ,"payload":"{\"_id\": {\"$oid\": \"6656b8fac0831732ce3027d4\"}}"}    

      Value (147 bytes):

      {"schema":\{"type":"string","optional":false}

      ,"payload":"{\"_id\":

      {\"$oid\": \"6656b8fac0831732ce3027d4\"}

      , \"order_id\": 100, \"item\": \"tea\"}"}
      -------------------------

      % Reached end of topic changelog.mongo.kafka.connect [0] at offset 4

      ```

            Assignee:
            Robert Walters (Inactive)
            Reporter:
            Roshan Joyce
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: