[KAFKA-113] Writemodel upsert strategy which can target multiple fields Created: 10/Jun/20  Updated: 27/Oct/23  Resolved: 08/Sep/20

Status: Closed
Project: Kafka Connector
Component/s: Sink
Affects Version/s: None
Fix Version/s: 1.3.0

Type: Improvement Priority: Major - P3
Reporter: Pat W Assignee: Ross Lawley
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Case:

 Description   

Hello again,

We recently upgraded our sharded mongodb cluster to 4.2.0+ and encountered an issue where we wanted upserts but the mongodb connector creates upserts which only match on the _id field and our sharded collections have compound shard keys.

I created a writemodel strategy which creates upserts that can be configured to target multiple fields, hopefully it can be of use. As before I have included tests,

https://github.com/mongodb/mongo-kafka/pull/16

Thanks!

-Pat



 Comments   
Comment by Ross Lawley [ 08/Sep/20 ]

Closing as works as designed for now but happy to review if there is a scenario where that wont be the case.

Comment by Ross Lawley [ 08/Sep/20 ]

In review you should be able to use the PartialValueStrategy to generate the shardkey match. See: writemodel strategy business keys.

KAFKA-155 highlights a bug where it is possible that the Replace strategy does not correctly match the compound key and that is due to be fixed in 1.3.0.

I think the combination of the PartialValueStrategy and the fix coming in KAFKA-155 should handle shard key / compound key scenarios. Let me know if that is not the case.

Ross

Generated at Thu Feb 08 09:05:37 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.