[KAFKA-394] Delete.on.null.values has no effect Created: 03/Jan/24  Updated: 04/Jan/24  Resolved: 04/Jan/24

Status: Closed
Project: Kafka Connector
Component/s: None
Affects Version/s: 1.11.1
Fix Version/s: None

Type: Question Priority: Unknown
Reporter: Fabian Bell Assignee: Ross Lawley
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Tombstone messages do not trigger a deletion even if `delete.on.null.values` is set to true. 

Configuration:

topics: "inputTopic"
connection.uri: "mongodb://blafoo:27017"
key.converter: "org.apache.kafka.connect.storage.StringConverter"
value.converter: "org.apache.kafka.connect.storage.StringConverter"
value.converter.schemas.enable: false
key.converter.schemas.enable: false
transforms: "PrepareId, PrepareValue"
transforms.PrepareId.type: "org.apache.kafka.connect.transforms.HoistField$Key"
transforms.PrepareId.field: "_id"
transforms.PrepareValue.type: "org.apache.kafka.connect.transforms.HoistField$Value"
transforms.PrepareValue.field: "value"
document.id.strategy: "com.mongodb.kafka.connect.sink.processor.id.strategy.ProvidedInKeyStrategy"
document.id.strategy.overwrite.existing: true
delete.on.null.values: true
database: "test"
collection: "test" 

How to reproduce:

 

  1. Send a kafka record with key: "test" and value: "test" to the inputTopic
    1. We see the following entry in MongoDB {"_id": "test, "value": "test"}
  2. Send a kafka record with key: "test" and value: null to the inputTopic 
    1. We see the following entry in MongoDB {"_id": "test, "value": null}

Expected behaviour:

After ** step 2 we would expect that we do not find any entry with {"_id": "test", ...} in MongoDB.



 Comments   
Comment by Ross Lawley [ 04/Jan/24 ]

Hi fabianx.bell@gmail.com,

Excellent, glad its working for you now.

Ross

Comment by Fabian Bell [ 03/Jan/24 ]

Hi ross@mongodb.com,

thank you for the quick response. You are right, the connector receives:

{"value": null}

I resolved this issue by adding:

transforms.PrepareValue.predicate: "isTombstone"
transforms.PrepareValue.negate: "true"
predicates: "isTombstone"
predicates.isTombstone.type: "org.apache.kafka.connect.transforms.predicates.RecordIsTombstone" 

 

Comment by Ross Lawley [ 03/Jan/24 ]

Hi fabianx.bell@gmail.com,

Are the SMTs are changing the value and making it not null? Specifically:

transforms.PrepareValue.type: "org.apache.kafka.connect.transforms.HoistField$Value"
transforms.PrepareValue.field: "value"

The connector only gets the record after the SMTs have been applied, so this could make it appear that sink record is not null when it originally was.

Ross

Comment by PM Bot [ 03/Jan/24 ]

Hi fabianx.bell@gmail.com, thank you for reporting this issue! The team will look into it and get back to you soon.

Generated at Thu Feb 08 09:06:16 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.