[KAFKA-68] MongoSourceConnector source record key is configurable Created: 10/Sep/19 Updated: 28/Oct/23 Resolved: 17/Aug/20 |
|
| Status: | Closed |
| Project: | Kafka Connector |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 1.3.0 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Wolfgang Strack | Assignee: | Ross Lawley |
| Resolution: | Fixed | Votes: | 2 |
| Labels: | sp-ga | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Epic Link: | Support schema for Source connector | ||||||||||||||||
| Description |
|
At time of this writing, when source records are produced to their respective topic, the message key is given by default in the format
Instead, it would be useful if there was an option to configure the message key so that, for example, you could choose a different field from the change-stream document to be converted to a source record. One particularly useful case would be to choose the "_id" of the actual collection document, if present in the |
| Comments |
| Comment by Ross Lawley [ 17/Aug/20 ] | ||||||||||
|
See: | ||||||||||
| Comment by Ross Lawley [ 05/Aug/20 ] | ||||||||||
|
The Schema can use any field from within the change stream document. For example the default key schema will be:
This will produce the '_id' field as a string. (If the field type is not a string, the JsonFormatter ( This approach will allow any value to become the key.
So I think the new Schema support for both the key and value will provide the required control over the SourceRecord key. Ross | ||||||||||
| Comment by Andrey B [ 02/Apr/20 ] | ||||||||||
|
Hi, @Ross Lawley Usecase the same as in ticket description. I need to configure the source record key. The simple case when I just need to write all events for the same document into the same partition. So I want to configure source record key as `documentKey._id`. But I can imagine when I need something else for the source record key. e.g. some field from `fullDocument`. So, it would be pretty useful even if you just add the possibility to use `documentKey._id` as source record key. | ||||||||||
| Comment by Ross Lawley [ 30/Mar/20 ] | ||||||||||
|
This is still on the backlog, we're still discussing how this would be configured. Please feel free to add your usecase? Would you want the _id field? Ross | ||||||||||
| Comment by Andrey B [ 26/Mar/20 ] | ||||||||||
|
Hello! | ||||||||||
| Comment by Seth Payne [ 24/Oct/19 ] | ||||||||||
|
In cases where the document key does not exist, we will use the resume token. | ||||||||||
| Comment by Ross Lawley [ 12/Sep/19 ] | ||||||||||
|
Hi wstrack@riffyn.com, no worries, I've updated the description. The ticket will be reviewed and scheduled in due course. Ross | ||||||||||
| Comment by Wolfgang Strack [ 11/Sep/19 ] | ||||||||||
|
Can't edit the description so I apologize for the mis-formatting. Also, for the use-case provided, I meant the "_id" would be pulled from the "documentKey" field of the change stream doc, rather than the potentially absent "fullDocument" field. |