[KAFKA-209] Projection always includes _id Created: 19/Mar/21 Updated: 28/Oct/23 Resolved: 23/Mar/21 |
|
| Status: | Closed |
| Project: | Kafka Connector |
| Component/s: | Sink |
| Affects Version/s: | None |
| Fix Version/s: | 1.5.0 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Robert Walters | Assignee: | Ross Lawley |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Description |
| Comments |
| Comment by Ross Lawley [ 23/Mar/21 ] | ||
|
After discussion with robert.walters, it was determined that projection should always be explicit. The current documentation supports this so it is surprising that _id was automatically included. | ||
| Comment by Githook User [ 23/Mar/21 ] | ||
|
Author: {'name': 'Ross Lawley', 'email': 'ross.lawley@gmail.com', 'username': 'rozza'}Message: Fixed `_id` projection. Previously, `_id` has always being projected even if not explicitly
| ||
| Comment by Ross Lawley [ 22/Mar/21 ] | ||
|
Hi juan.soto, Yes the bug is in the field projection - which previously never removed the _id field under any circumstances. So if you happened to be using two separate systems and each contained their own _id values the connector could never couple them up even if the business logic is kept in a separate key (eg. order-id). Now _id will only be included/excluded by the projection if it is explicitly added Ross | ||
| Comment by Juan Soto (Inactive) [ 22/Mar/21 ] | ||
|
Hi!! Then, is the a bug for all "OneBusinessKeyStrategy" operation ? is not just for the delete? Regards, Juan | ||
| Comment by Ross Lawley [ 22/Mar/21 ] | ||
|
After discussion with robert.walters we've come to the conclusion that it is a bug to automatically include the _id field if you are explicitly listing the fields to be projected. Its surprising and undocumented behaviour. | ||
| Comment by Ross Lawley [ 22/Mar/21 ] | ||
|
Hi robert.walters, OK this behaviour is expected from the terms of the connector. So a couple of points that need clarifying as to why that is: 1) The document.id.strategy will always include a _id field if it exists in the sink record. So to include order-id as part of the filter to be removed you must set:
That will then make the following query for the DeleteOneModel:
I've added a logging message if the _id field exists but document.id.strategy.overwrite.existing is false. Please file any documentation tickets that are needed to improve the documentation. Ross | ||
| Comment by Githook User [ 22/Mar/21 ] | ||
|
Author: {'name': 'Ross Lawley', 'email': 'ross.lawley@gmail.com', 'username': 'rozza'}Message: Improve DocumentIdAdder logging. Log a warning message when there the `_id` value and the
|