[KAFKA-165] Support new updateDescription field truncatedArrays and dot format for updatedFields Created: 09/Oct/20  Updated: 07/Feb/24

Status: Backlog
Project: Kafka Connector
Component/s: Source
Affects Version/s: None
Fix Version/s: 1.12.0

Type: Improvement Priority: Minor - P4
Reporter: Ross Lawley Assignee: Unassigned
Resolution: Unresolved Votes: 4
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-47354 Implement change stream oplog parsing... Closed
Quarter: FY23Q4
Case:

 Description   

New fields have been added to updatedDescription:

updateDescription: {
  updatedFields: {},
  truncatedArrays: [{field: "arrayField", newSize: 2}]  // New field.
  disambiguatedPaths: { "residences.0.0": [ "residences", 0, "0" ] } // New since MongoDB 6.0
}

Along with the truncatedArrays field comes partial updates in the updatedFields document.

The original complexity is this add ambiguity to the updateFields paths where an update to "a.b.0.c.1.d" could update either:

{ a:  {b: [{c: [<val0>, {d: "field_to_be_updated"}, …]}, …]} }
 
// Or
{ "a.b": {"0": {"c.1": {d: "field_to_be_updated"}}}

The syntax required for the update is different and currently not supported. As such, the existing code checks for unsupported updateDescription fields and throws an unsupported exception - users are required to use fullDocument

However, Since Mongo 6.0 expanded events have been supported. This adds disambiguatedPaths to the Update Event.

Adding a new configuration to the connector could allow for supporting expanded events, this would allow the connector to support both truncatedArrays and partial updates.



 Comments   
Comment by Vittal Pai [ 31/May/22 ]

robert.walters@mongodb.com, Do we have any ETA for this work item? The solution which we are building for one of our ISV partners is blocked due to this as we are not able to use CDC handlers for update operations with MongoDB ver 5.0.  More details: HELP-32826

 

Generated at Thu Feb 08 09:05:44 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.