<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 09:05:34 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[KAFKA-97] Configuration for MongoDB Kafka Connect Connector to avoid {&quot;_id&quot;: {&quot;$numberLong&quot;: &quot;11111111&quot;}</title>
                <link>https://jira.mongodb.org/browse/KAFKA-97</link>
                <project id="16285" key="KAFKA">Kafka Connector</project>
                    <description>&lt;p&gt;I am using the attached configuration on the source side and I am seeing the following in the Kafka topic:&lt;/p&gt;

&lt;p&gt;{&lt;tt&gt;}&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;&quot;somefield&quot;: {&lt;/tt&gt;&lt;tt&gt;&quot;$numberLong&quot;: &quot;2342423432432432434324&quot;&lt;/tt&gt;&lt;tt&gt;}&lt;/tt&gt;&lt;br/&gt;
&#160;&lt;br/&gt;
is there a way we can have the data formatted like this:&#160;&lt;tt&gt;&quot;somefield&quot;: 2342423432432432434324&quot;.&lt;/tt&gt;&lt;br/&gt;
&#160;&lt;br/&gt;
Attached config uses&#160;StringConverter, but I have tried JSON as well as Avro with the following configuration&lt;br/&gt;
&#160;&lt;br/&gt;
&quot;key.converter&quot;:&quot;io.confluent.connect.avro.AvroConverter&quot;,&lt;br/&gt;
&quot;key.converter.schema.registry.url&quot;:&quot;http://schema-registry:8081&quot;,&lt;br/&gt;
&quot;value.converter&quot;: &quot;io.confluent.connect.avro.AvroConverter&quot;,&lt;br/&gt;
&quot;value.converter.schema.registry.url&quot;:&quot;http://schema-registry:8081&quot;&lt;br/&gt;
&#160;&lt;br/&gt;
Thanks in advance.&lt;br/&gt;
&#160;&lt;br/&gt;
&#160;&lt;br/&gt;
{&lt;tt&gt;}&lt;/tt&gt;&lt;/p&gt;</description>
                <environment>MongoDB: 3.6.17, Kafka Connector 1.1</environment>
        <key id="1309919">KAFKA-97</key>
            <summary>Configuration for MongoDB Kafka Connect Connector to avoid {&quot;_id&quot;: {&quot;$numberLong&quot;: &quot;11111111&quot;}</summary>
                <type id="2" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14711&amp;avatarType=issuetype">New Feature</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="-1">Unassigned</assignee>
                                    <reporter username="sabari.mgn@gmail.com">Sabari Gandhi</reporter>
                        <labels>
                    </labels>
                <created>Thu, 9 Apr 2020 18:07:49 +0000</created>
                <updated>Wed, 17 Jun 2020 14:32:21 +0000</updated>
                            <resolved>Wed, 17 Jun 2020 14:32:21 +0000</resolved>
                                    <version>1.1</version>
                                                                        <votes>2</votes>
                                    <watches>2</watches>
                                                                                                                <comments>
                            <comment id="3212247" author="ross@10gen.com" created="Wed, 17 Jun 2020 14:32:21 +0000"  >&lt;p&gt;Will fix as part of &lt;a href=&quot;https://jira.mongodb.org/browse/KAFKA-99&quot; title=&quot;Support relaxed Json format in the source connector&quot; class=&quot;issue-link&quot; data-issue-key=&quot;KAFKA-99&quot;&gt;&lt;del&gt;KAFKA-99&lt;/del&gt;&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3035634" author="sabari.mgn@gmail.com" created="Fri, 10 Apr 2020 21:23:56 +0000"  >&lt;p&gt;Update: When I further looked into Kafka code I do see the validation that happens on the message for the transformations.&#160;&#160;&lt;a href=&quot;https://github.com/a0x8o/kafka/blob/master/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/util/Requirements.java#L52&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/a0x8o/kafka/blob/master/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/util/Requirements.java#L52.&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Transformations expect the schema type to have a struct with relevant data but Mongo source connector doesn&apos;t produce the struct format when it publishes messages to Kafka topic. Or is there any way to enable the configuration? Thanks in advance&lt;/p&gt;</comment>
                            <comment id="3035596" author="sabari.mgn@gmail.com" created="Fri, 10 Apr 2020 20:54:40 +0000"  >&lt;p&gt;To Summarize: I need a clean JSON in my S3. If I have&#160;StringConverter in source config and&#160;JsonConverter on the sink config I have clean JSON output but if I insert something like&#160;&#160;db.investigate1.insert({&quot;_id&quot; : NumberLong(&quot;298384262959477&quot;),&quot;name&quot; : &quot;sample&quot;, &quot;type&quot; : &quot;pipeline&quot;})&lt;/p&gt;

&lt;p&gt;the output I see in S3 is as follows:&#160;&lt;br/&gt;
{&quot;fullDocument&quot;:{&quot;name&quot;:&quot;sample&quot;,&quot;_id&quot;:&lt;/p&gt;
{&quot;$numberLong&quot;:&quot;298384262959477&quot;}
&lt;p&gt;,&quot;type&quot;:&quot;pipeline&quot;},&quot;ns&quot;:{&quot;coll&quot;:&quot;investigate1&quot;,&quot;db&quot;:&quot;test&quot;},&quot;documentKey&quot;:{&quot;_id&quot;:{&quot;$numberLong&quot;:&quot;298384262959477&quot;}},&quot;operationType&quot;:&quot;insert&quot;,&quot;_id&quot;:{&quot;_data&quot;:&lt;/p&gt;
{&quot;$binary&quot;:&quot;gl6Qw3cAAAADRh5faWQAMQIewgAAIuoAWhAEs+M48TaZTnOzj+bC1yVP2QQ=&quot;,&quot;$type&quot;:&quot;00&quot;}
&lt;p&gt;,&quot;_typeBits&quot;:{&quot;$binary&quot;:&quot;Ag==&quot;,&quot;$type&quot;:&quot;00&quot;}}}&lt;br/&gt;
&#160;&lt;br/&gt;
I would like to see something like&#160;&quot;fullDocument&quot;:{&quot;name&quot;:&quot;sample&quot;,&quot;_id&quot;:298384262959477, &quot;type&quot;:&quot;pipeline&quot;}... in my S3.&lt;br/&gt;
&#160;&lt;br/&gt;
I also looked into transforms that are available on the sink config. But I see the transforms does not work here as it works for debezium connector. When i looked into it more I doubt the message format produced by source connector in kafka topic is not similar to debezium. Do we have transform support for source connector. Please advice.&lt;br/&gt;
&#160;&lt;/p&gt;</comment>
                            <comment id="3034624" author="sabari.mgn@gmail.com" created="Fri, 10 Apr 2020 12:57:19 +0000"  >&lt;p&gt;We sink messages from Kafka topic to S3 and we can&apos;t run Athena queries because it breaks on the $. Also FYI, I have attached the sink configuration in the ticket.&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Also, I tried transformations on S3 sink configurations it doesn&apos;t readily work with Mongo source connector. Is it because of the message format that&apos;s produced by the source connector to the Kafka? Can we please get some information on transformation supported by Mongo source connector? I confirmed some transformations work for debezium hence I was asking about the&#160;message format produced to the Kafka. Thanks&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="1324212">KAFKA-99</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="255699" name="sink_config.json" size="2821" author="sabari.mgn@gmail.com" created="Fri, 10 Apr 2020 12:57:28 +0000"/>
                            <attachment id="255621" name="source_config.json" size="976" author="sabari.mgn@gmail.com" created="Thu, 9 Apr 2020 18:07:33 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hx1ye7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                </customfields>
    </item>
</channel>
</rss>