<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 09:05:30 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[KAFKA-68] MongoSourceConnector source record key is configurable</title>
                <link>https://jira.mongodb.org/browse/KAFKA-68</link>
                <project id="16285" key="KAFKA">Kafka Connector</project>
                    <description>&lt;p&gt;At time of this writing, when source records are produced to their respective topic, the message key is given by default in the format&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{  &quot;_id&quot;: { &amp;lt;bson resume token&amp;gt; }} &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Instead, it would be useful if there was an option to configure the message key so that, for example, you could choose a different field from the change-stream document to be converted to a source record.&lt;/p&gt;

&lt;p&gt;One particularly useful case would be to choose the &quot;_id&quot; of the actual collection document, if present in the &lt;del&gt;&quot;fullDocument&quot;&lt;/del&gt; &quot;documentKey&quot; field. This would allow the corresponding kafka topic to be partitioned on that collection&apos;s &quot;_id&quot; field.&lt;/p&gt;</description>
                <environment></environment>
        <key id="922947">KAFKA-68</key>
            <summary>MongoSourceConnector source record key is configurable</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="ross@mongodb.com">Ross Lawley</assignee>
                                    <reporter username="wstrack@riffyn.com">Wolfgang Strack</reporter>
                        <labels>
                            <label>sp-ga</label>
                    </labels>
                <created>Tue, 10 Sep 2019 23:50:52 +0000</created>
                <updated>Sat, 28 Oct 2023 10:46:23 +0000</updated>
                            <resolved>Mon, 17 Aug 2020 14:56:53 +0000</resolved>
                                                    <fixVersion>1.3.0</fixVersion>
                                                        <votes>2</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="3342494" author="ross@10gen.com" created="Mon, 17 Aug 2020 14:56:53 +0000"  >&lt;p&gt;See: &lt;a href=&quot;https://jira.mongodb.org/browse/KAFKA-124&quot; title=&quot;Support output to schema&quot; class=&quot;issue-link&quot; data-issue-key=&quot;KAFKA-124&quot;&gt;&lt;del&gt;KAFKA-124&lt;/del&gt;&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3322794" author="ross@10gen.com" created="Wed, 5 Aug 2020 15:00:12 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/KAFKA-124&quot; title=&quot;Support output to schema&quot; class=&quot;issue-link&quot; data-issue-key=&quot;KAFKA-124&quot;&gt;&lt;del&gt;KAFKA-124&lt;/del&gt;&lt;/a&gt; will allow users to define the schema for the key / record values, using Avro schema definitions.&lt;/p&gt;

&lt;p&gt;The Schema can use any field from within the change stream document.   For example the default key schema will be:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  &quot;type&quot;: &quot;record&quot;,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  &quot;name&quot;: &quot;keySchema&quot;,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  &quot;fields&quot; : [{&quot;name&quot;: &quot;_id&quot;, &quot;type&quot;: &quot;string&quot;}]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;This will produce the &apos;_id&apos; field as a string. (If the field type is not a string, the JsonFormatter (&lt;a href=&quot;https://jira.mongodb.org/browse/KAFKA-99&quot; title=&quot;Support relaxed Json format in the source connector&quot; class=&quot;issue-link&quot; data-issue-key=&quot;KAFKA-99&quot;&gt;&lt;del&gt;KAFKA-99&lt;/del&gt;&lt;/a&gt;) will be used to convert it to a string).&lt;/p&gt;

&lt;p&gt;This approach will allow any value to become the key.  &lt;a href=&quot;https://jira.mongodb.org/browse/KAFKA-137&quot; title=&quot;Add support for dotted field names.&quot; class=&quot;issue-link&quot; data-issue-key=&quot;KAFKA-137&quot;&gt;&lt;del&gt;KAFKA-137&lt;/del&gt;&lt;/a&gt; will allow for shortened schemas as it will support dotted lookups. For example the below avro schema definition would use the &lt;tt&gt;fullDocument.partitionKey&lt;/tt&gt; field and if that field is missing it will use the default value of &lt;tt&gt;DefaultPartition&lt;/tt&gt;.&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  &quot;type&quot;: &quot;record&quot;,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  &quot;name&quot;: &quot;keySchema&quot;,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  &quot;fields&quot; : [{&quot;name&quot;: &quot;fullDocument.partitionKey&quot;, &quot;type&quot;: &quot;string&quot;, &quot;default&quot;: &quot;DefaultPartition&quot;}]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;So I think the new Schema support for both the key and value will provide the required control over the SourceRecord key.&lt;/p&gt;

&lt;p&gt;Ross&lt;/p&gt;</comment>
                            <comment id="3022880" author="JIRAUSER1269378" created="Thu, 2 Apr 2020 12:40:40 +0000"  >&lt;p&gt;Hi, @Ross Lawley&lt;/p&gt;

&lt;p&gt;Usecase the same as in ticket description.&lt;/p&gt;

&lt;p&gt;I need to configure the source record key.&lt;/p&gt;

&lt;p&gt;The simple case when I just need to write all events for the same document into the same partition. So I want to configure source record key as `documentKey._id`.&lt;/p&gt;

&lt;p&gt;But I can imagine when I need something else for the source record key. e.g. some field from `fullDocument`.&lt;/p&gt;

&lt;p&gt;So, it would be pretty useful even if you just add the possibility to use `documentKey._id` as source record key.&lt;/p&gt;</comment>
                            <comment id="3012335" author="ross@10gen.com" created="Mon, 30 Mar 2020 07:48:20 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=andreworty%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;andreworty@gmail.com&quot;&gt;andreworty@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;This is still  on the backlog, we&apos;re still discussing how this would be configured. Please feel free to add your usecase?  Would you want the _id field?&lt;/p&gt;

&lt;p&gt;Ross&lt;/p&gt;</comment>
                            <comment id="3001892" author="JIRAUSER1269378" created="Thu, 26 Mar 2020 10:11:10 +0000"  >&lt;p&gt;Hello!&lt;br/&gt;
Could someone tell in which status is this ticket?&#160;&lt;/p&gt;</comment>
                            <comment id="2499377" author="seth.payne" created="Thu, 24 Oct 2019 15:42:17 +0000"  >&lt;p&gt;In cases where the document key does not exist, we will use the resume token.&lt;/p&gt;</comment>
                            <comment id="2417499" author="ross@10gen.com" created="Thu, 12 Sep 2019 08:36:26 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=wstrack%40riffyn.com&quot; class=&quot;user-hover&quot; rel=&quot;wstrack@riffyn.com&quot;&gt;wstrack@riffyn.com&lt;/a&gt;, no worries, I&apos;ve updated the description. The ticket will be reviewed and scheduled in due course.&lt;/p&gt;

&lt;p&gt;Ross&lt;/p&gt;</comment>
                            <comment id="2416918" author="wstrack@riffyn.com" created="Wed, 11 Sep 2019 18:21:02 +0000"  >&lt;p&gt;Can&apos;t edit the description so I apologize for the mis-formatting. Also, for the use-case provided, I meant the &quot;_id&quot; would be pulled from the &quot;documentKey&quot; field of the change stream doc, rather than the potentially absent &quot;fullDocument&quot; field.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="1406339">KAFKA-123</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="1430774">KAFKA-137</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>KAFKA-40</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hve8hr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                </customfields>
    </item>
</channel>
</rss>