<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 09:05:40 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[KAFKA-140] Add behavior.on.malformed.documents config to Sink Connector</title>
                <link>https://jira.mongodb.org/browse/KAFKA-140</link>
                <project id="16285" key="KAFKA">Kafka Connector</project>
                    <description>&lt;p&gt;When we send data from Kafka to MongoDB using the Sink Connector, MongoDB automatically detects the data type for each field. So far so good.&lt;/p&gt;

&lt;p&gt;When the Kafka payload on the topic changes - meaning that an existing field suddenly has a new data type - the sink connector fails.&#160;&#160;&lt;/p&gt;

&lt;p&gt;This is not in any way an unexpected behavior.&#160;&lt;/p&gt;

&lt;p&gt;However, an erroneous message, in Kafka terminology referred to as a poison pill, ruins the topic/sink connector combination. At a connector restart the latest offset is the new starting point and the connector immediately fails again on the same message.&lt;/p&gt;

&lt;p&gt;Changing the consumer-group offset is the only way to work around this. Nice on a testing environment, but unacceptable in production.&#160;&#160;&lt;/p&gt;

&lt;p&gt;Extensive documentation is dedicated to configuring&#160;errors.tolerance (&lt;a href=&quot;https://docs.mongodb.com/kafka-connector/master/kafka-sink-properties/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://docs.mongodb.com/kafka-connector/master/kafka-sink-properties/&lt;/a&gt;). However and unfortunately, the generic error.tolerance connector settings only apply to the (de)serialization and SMT phase, and not to the actual connector code itself. In our case, and potentially many others, we are powerless when it comes to dealing with exceptions in the actual sink phase of the connector.&#160;&#160;&lt;/p&gt;

&lt;p&gt;What would be highly desirable is a configuration property equal to ElastisSearch sink connector&apos;s behavior.on.malformed.documents&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://docs.confluent.io/current/connect/kafka-connect-elasticsearch/configuration_options.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://docs.confluent.io/current/connect/kafka-connect-elasticsearch/configuration_options.html&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Are there any plans already for such a setting?&lt;/p&gt;</description>
                <environment></environment>
        <key id="1437641">KAFKA-140</key>
            <summary>Add behavior.on.malformed.documents config to Sink Connector</summary>
                <type id="2" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14711&amp;avatarType=issuetype">New Feature</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="ross@mongodb.com">Ross Lawley</assignee>
                                    <reporter username="jeffrey.vanhelden@thewarehouse.co.nz">J vH</reporter>
                        <labels>
                    </labels>
                <created>Tue, 11 Aug 2020 01:00:38 +0000</created>
                <updated>Mon, 21 Sep 2020 13:10:18 +0000</updated>
                            <resolved>Mon, 21 Sep 2020 13:10:12 +0000</resolved>
                                    <version>1.2.0</version>
                                    <fixVersion>1.3.0</fixVersion>
                                    <component>Sink</component>
                                        <votes>0</votes>
                                    <watches>2</watches>
                                                                                                                <comments>
                            <comment id="3402972" author="ross@10gen.com" created="Mon, 21 Sep 2020 13:10:12 +0000"  >&lt;p&gt;Reusing the existing errors.tolerance setting will handle invalid records / poison pills&lt;/p&gt;</comment>
                            <comment id="3332022" author="ross@10gen.com" created="Tue, 11 Aug 2020 08:03:22 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jeffrey.vanhelden%40thewarehouse.co.nz&quot; class=&quot;user-hover&quot; rel=&quot;jeffrey.vanhelden@thewarehouse.co.nz&quot;&gt;jeffrey.vanhelden@thewarehouse.co.nz&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thanks for the ticket, there is an epic for improving error handling, which I&apos;ve linked.  This is probably a duplicate of &lt;a href=&quot;https://jira.mongodb.org/browse/KAFKA-105&quot; title=&quot;Support errors.tolerance&quot; class=&quot;issue-link&quot; data-issue-key=&quot;KAFKA-105&quot;&gt;&lt;del&gt;KAFKA-105&lt;/del&gt;&lt;/a&gt;, but I&apos;ll keep open to ensure the poison pill scenario is covered.&lt;/p&gt;

&lt;p&gt;Ross&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="1341837">KAFKA-105</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>KAFKA-134</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hxmcin:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                </customfields>
    </item>
</channel>
</rss>