Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-209

Projection always includes _id

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 1.5.0
    • Affects Version/s: None
    • Component/s: Sink
    • None

      Problem Description

      DeleteOneBusinessKeyStrategy throws an error when doing partial projection and an existing _id field exists and document.id.strategy.overwrite.existing is not set or is false.
      Also with document.id.strategy.overwrite.existing: true the _id field is included even though it is not listed in the projection list.

      Steps to Reproduce

      I have a kafkatopic "FaceMaskWeb.OrderCancel" that has a message:

      "{\"_id\": {\"$oid\": \"6053bb753283c9f1e8584cb4\"}, \"order-id\": 100}"
      

      I have a sink that is defined on the "FaceMaskWeb.OrderCancel" topic as

      "database":"FaceMaskWeb",
          "collection":"Orders",
          "writemodel.strategy": "com.mongodb.kafka.connect.sink.writemodel.strategy.DeleteOneBusinessKeyStrategy",
          "document.id.strategy": "com.mongodb.kafka.connect.sink.processor.id.strategy.PartialValueStrategy",
          "document.id.strategy.partial.value.projection.type": "AllowList",
          "document.id.strategy.partial.value.projection.list": "order-id",
      

      I am trying to delete the document in the Orders collection that looks like this:

        {
          _id: ObjectId("60538d8d360dc9fda35f1312"),
          'customer-id': 123,
          'order-id': 100,
          order: { lineitem: 1, SKU: 'FACE1', quantity: 1 }
        }
      

      My Sink fails with:

      org.apache.kafka.connect.errors.DataException: Could not build the WriteModel, the value document does not contain an _id field of type BsonDocument which holds the business key fields.
      

      I am trying to delete the document based upon the order-id field not any _id field, yet this error message makes me think that the order-id field need to be apart of the _id field maybe as a compound key?

      From a slack discussion with Ross, the document.id.strategy.partial.value.projection.list might not be returning a document in this scenario where there is just one key in the project list.

      Expected Results

      field in allowlist would be used as the field to delete in the collection

      Actual Results

      Additional Notes

            Assignee:
            ross@mongodb.com Ross Lawley
            Reporter:
            robert.walters@mongodb.com Robert Walters (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: