<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 09:05:46 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[KAFKA-177] MongoDB Sink Connector deadletterqueuetopic issue</title>
                <link>https://jira.mongodb.org/browse/KAFKA-177</link>
                <project id="16285" key="KAFKA">Kafka Connector</project>
                    <description>&lt;p&gt;Our development team has found a defect on the Kafka Connect MongoDB Sink (&lt;a href=&quot;https://docs.mongodb.com/kafka-connector/master/kafka-sink&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://docs.mongodb.com/kafka-connector/master/kafka-sink&lt;/a&gt;). It seems that support for the deadletterqueue has been removed/disappeared and cant get the feature to working in the latest version as per the MongoDB documentation. (we also confirmed the code is missing from the mainline in GIT)&lt;/p&gt;

&lt;p&gt;According this document, the sink connector is supposed to support deadletterqueue topic (&lt;a href=&quot;https://docs.mongodb.com/kafka-connector/master/kafka-sink-properties#dead-letter-queue-configuration-settings&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://docs.mongodb.com/kafka-connector/master/kafka-sink-properties#dead-letter-queue-configuration-settings&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;errors.deadletterqueue.topic.name=example.deadletterqueue&lt;/p&gt;

&lt;p&gt;However, when I publish a message that violates the MongoDB unique index, I do not see the message in dead letter queue topic. I tried both console consumer and in Spring Boot application, no luck.&lt;/p&gt;

&lt;p&gt;And then, I went to the GitHub (master) to check the source code. It seems the dead letter queue topic is not part of MongoSinkTopicConfig.java (&lt;a href=&quot;https://github.com/mongodb/mongo-kafka/blob/master/src/main/java/com/mongodb/kafka/connect/sink/MongoSinkTopicConfig.java&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo-kafka/blob/master/src/main/java/com/mongodb/kafka/connect/sink/MongoSinkTopicConfig.java&lt;/a&gt;), i.e. errors.deadletterqueue.topic.name is not part of the configuration. On the other hand, that properties is available in MongoSourceConfig.java (&lt;a href=&quot;https://github.com/mongodb/mongo-kafka/blob/master/src/main/java/com/mongodb/kafka/connect/source/MongoSourceConfig.java&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo-kafka/blob/master/src/main/java/com/mongodb/kafka/connect/source/MongoSourceConfig.java&lt;/a&gt;).&lt;/p&gt;</description>
                <environment>Dev</environment>
        <key id="1553401">KAFKA-177</key>
            <summary>MongoDB Sink Connector deadletterqueuetopic issue</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13202">Works as Designed</resolution>
                                        <assignee username="ross@mongodb.com">Ross Lawley</assignee>
                                    <reporter username="adam.cowin@td.com">Adam Cowin</reporter>
                        <labels>
                    </labels>
                <created>Thu, 26 Nov 2020 22:27:28 +0000</created>
                <updated>Fri, 27 Oct 2023 11:54:14 +0000</updated>
                            <resolved>Fri, 27 Nov 2020 09:58:19 +0000</resolved>
                                    <version>1.3.0</version>
                                                    <component>Documentation</component>
                    <component>Sink</component>
                                        <votes>0</votes>
                                    <watches>3</watches>
                                                                                                                <comments>
                            <comment id="3507358" author="ross@10gen.com" created="Fri, 27 Nov 2020 16:17:52 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ming.li2%40td.com&quot; class=&quot;user-hover&quot; rel=&quot;ming.li2@td.com&quot;&gt;ming.li2@td.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;The dead letter queue is really a feature of Kafka Connect and not of the connector implementations.  It&apos;s an abstraction at a higher level and as such is not controlled by the connectors.&lt;/p&gt;

&lt;p&gt;Ross&lt;/p&gt;</comment>
                            <comment id="3507353" author="JIRAUSER1256832" created="Fri, 27 Nov 2020 16:12:36 +0000"  >&lt;p&gt;Hi Ross,&#160;&lt;/p&gt;

&lt;p&gt;Thank you for the explanation. From the table that you provided, it means both source and sink connectors cannot publish messages to dead letter queue due to the nature of Apache Kafka connector?&lt;/p&gt;

&lt;p&gt;Regards,&lt;/p&gt;

&lt;p&gt;Ming Li&lt;/p&gt;</comment>
                            <comment id="3507057" author="ross@10gen.com" created="Fri, 27 Nov 2020 09:58:06 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=adam.cowin%40td.com&quot; class=&quot;user-hover&quot; rel=&quot;adam.cowin@td.com&quot;&gt;adam.cowin@td.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thanks for the ticket.  The dead letter queue functionality is provided by Kafka connect and not the actual connectors:  See: &lt;a href=&quot;https://www.confluent.io/blog/kafka-connect-deep-dive-error-handling-dead-letter-queues/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://www.confluent.io/blog/kafka-connect-deep-dive-error-handling-dead-letter-queues/&lt;/a&gt;&lt;br/&gt;
for more information about dead letter queues.&lt;/p&gt;

&lt;p&gt;The documentation contains a reference to the main kafka dead letter queue settings for users.  The following table is what Kafka connect supports with regards to dead letter queues.&lt;/p&gt;

&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;Connector lifecycle stage &lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt; Description &lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt; Handled? &lt;/th&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt; start &lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt; When a connector first starts, it will perform the required initialization such as connecting to the datastore &lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt; No &lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt; poll (for source connector) &lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt; Read records from the source datastore &lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt; No &lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt; convert &lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt; Read/write data from/to the Kafka topic and &lt;span class=&quot;error&quot;&gt;&amp;#91;de&amp;#93;&lt;/span&gt;serialize the JSON/Avro, etc. &lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt; Yes &lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt; transform &lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt; Apply any configured Single Message Transform &lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt; Yes &lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt; put (for sink connector) &lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt; Write the records to the target datastore &lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt; No &lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;


&lt;p&gt;The sink connector cannot publish messages to the dead letter queue itself, but it does respect the &lt;tt&gt;errors.log.enable&lt;/tt&gt; and &lt;tt&gt;errors.tolerance&lt;/tt&gt; settings. Note: for the source connector, we added similar functionality to the dead letter queue that mimics the sink version. This is used when change stream messages cannot be converted to the set schema.&lt;/p&gt;

&lt;p&gt;For future reference, for questions like these, I wanted to give you some resources to get this question answered more quickly:&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Our MongoDB support portal, located at &lt;a href=&quot;https://support.mongodb.com/welcome&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;support.mongodb.com&lt;/a&gt;&lt;/li&gt;
	&lt;li&gt;Our MongoDB community portal, located &lt;a href=&quot;https://developer.mongodb.com/community/forums/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;&lt;/li&gt;
	&lt;li&gt;If you are an Atlas customer, there is free support offered 24/7 in the lower right hand corner of the UI&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Just in case you have already opened a support case and are not receiving sufficient help, please let me know and I can facilitate escalating your issue.&lt;/p&gt;

&lt;p&gt;Thank you!&lt;/p&gt;

&lt;p&gt;Ross&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="1667744">KAFKA-215</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                        <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[5002K00000r3BsvQAE]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hy59in:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                </customfields>
    </item>
</channel>
</rss>