<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 09:06:05 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[KAFKA-311] Failed to resume change stream - Bad resume token</title>
                <link>https://jira.mongodb.org/browse/KAFKA-311</link>
                <project id="16285" key="KAFKA">Kafka Connector</project>
                    <description>&lt;p&gt;Hi. I am deploying latest version 0.28.0 of strimzi kafka operator helm chart in my Kubernetes cluster (&lt;a href=&quot;https://artifacthub.io/packages/helm/strimzi/strimzi-kafka-operator&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://artifacthub.io/packages/helm/strimzi/strimzi-kafka-operator&lt;/a&gt;) with mongodb sink connector to stream data from Kafka topic to MongoDb database.&lt;/p&gt;

&lt;p&gt;I get the same error when new data is added to Kafka topic. I get this error in the logs:&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&lt;tt&gt;2022-04-21 12:23:20,614 INFO &lt;span class=&quot;error&quot;&gt;&amp;#91;mongodb-source-connector|task-0&amp;#93;&lt;/span&gt; Watching for collection changes on &apos;weather.data&apos; (com.mongodb.kafka.connect.source.MongoSourceTask) &lt;span class=&quot;error&quot;&gt;&amp;#91;task-thread-mongodb-source-connector-0&amp;#93;&lt;/span&gt;&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;2022-04-21 12:23:20,615 INFO &lt;span class=&quot;error&quot;&gt;&amp;#91;mongodb-source-connector|task-0&amp;#93;&lt;/span&gt; Resuming the change stream after the previous offset using resumeAfter (com.mongodb.kafka.connect.source.MongoSourceTask) &lt;span class=&quot;error&quot;&gt;&amp;#91;task-thread-mongodb-source-connector-0&amp;#93;&lt;/span&gt;&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;&lt;b&gt;2022-04-21 12:23:20,617 INFO &lt;span class=&quot;error&quot;&gt;&amp;#91;mongodb-source-connector|task-0&amp;#93;&lt;/span&gt; Failed to resume change stream: Bad resume token: _data of missing or of wrong type{_id: 5553a998e4b02cf715119f6e, copyingData: true} 40647 (com.mongodb.kafka.connect.source.MongoSourceTask) &lt;span class=&quot;error&quot;&gt;&amp;#91;task-thread-mongodb-source-connector-0&amp;#93;&lt;/span&gt;&lt;/b&gt;&lt;/tt&gt;&lt;/p&gt;</description>
                <environment></environment>
        <key id="2030563">KAFKA-311</key>
            <summary>Failed to resume change stream - Bad resume token</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13202">Works as Designed</resolution>
                                        <assignee username="ross@mongodb.com">Ross Lawley</assignee>
                                    <reporter username="abdul.basith.kj@gmail.com">Abdul Basith</reporter>
                        <labels>
                            <label>external-user</label>
                    </labels>
                <created>Thu, 21 Apr 2022 21:58:19 +0000</created>
                <updated>Fri, 27 Oct 2023 11:54:11 +0000</updated>
                            <resolved>Tue, 26 Apr 2022 12:56:59 +0000</resolved>
                                                                                        <votes>0</votes>
                                    <watches>2</watches>
                                                                                                                <comments>
                            <comment id="4514766" author="JIRAUSER1269336" created="Thu, 28 Apr 2022 21:34:36 +0000"  >&lt;p&gt;Thanks much Ross. Appreciate the support provided.&lt;/p&gt;</comment>
                            <comment id="4506259" author="ross@10gen.com" created="Tue, 26 Apr 2022 12:56:43 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=abdul.basith.kj%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;abdul.basith.kj@gmail.com&quot;&gt;abdul.basith.kj@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;blockquote&gt;&lt;p&gt;The problem arises only when I use the Python API to add new data directly to the topic which could have been a rogue record causing that issue. I removed that record and I don&apos;t get that issue anymore.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;Glad you were able to sort the issue. From the error message it looks like some data was projected out that was required by the source connector (possibly by the provided pipeline).  There are expectations regarding the data created by the source connector. Also error handling by the sink connector can be done via a DLQ  - see the errors.tolerance Kafka configuration.&lt;/p&gt;


&lt;p&gt;Just to let you know for general usage questions our MongoDB community portal, located &lt;a href=&quot;https://developer.mongodb.com/community/forums/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt; might be useful as you can discuss any issues, how to&apos;s with the wider community.&lt;/p&gt;

&lt;p&gt;All the best,&lt;/p&gt;

&lt;p&gt;Ross Lawley&lt;/p&gt;

&lt;p&gt;Ross&lt;/p&gt;



</comment>
                            <comment id="4502330" author="JIRAUSER1269336" created="Sun, 24 Apr 2022 22:09:58 +0000"  >&lt;p&gt;Hi. I have found something.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;I have both Mongo source and sink connectors enabled. Now when I add a document directly via a tool like &apos;Studio 3T&apos; by using Copy document and Paste document after changing the Id, the new record gets copied to the topic through source connector and then it&apos;s copied to the sink Mongo database through sink connector - the whole workflow works excellent.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;The problem arises only when I use the Python API to add new data directly to the topic which could have been a rogue record causing that issue. I removed that record and I don&apos;t get that issue anymore.&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Now when I try to add a new record with the below code, the record does not get copied to the topic and I have a new error below:&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;&lt;tt&gt;from kafka import KafkaProducer&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;from kafka.errors import KafkaError&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;&lt;tt&gt;import json&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&lt;tt&gt;producer = KafkaProducer(bootstrap_servers=&lt;span class=&quot;error&quot;&gt;&amp;#91;&amp;quot;broker_ip:9092&amp;quot;&amp;#93;&lt;/span&gt;, value_serializer=lambda m: json.dumps(m).encode(&apos;ascii&apos;))&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;{{producer.send(&apos;kafka.weather.data&apos;, {&quot;key&quot;:&quot;_id&quot;, &quot;value&quot;:&quot;{\&quot;_id\&quot;: &lt;/p&gt;
&lt;div class=&quot;error&quot;&gt;&lt;span class=&quot;error&quot;&gt;Unknown macro: {&amp;quot;$oid&amp;quot;}&lt;/span&gt; &lt;/div&gt;
&lt;p&gt;, \&quot;st\&quot;: \&quot;x+47100-059501\&quot;, \&quot;ts\&quot;: {\&quot;$date\&quot;: 447400700000}, \&quot;position\&quot;: {\&quot;type\&quot;: \&quot;Point\&quot;, \&quot;coordinates\&quot;: &lt;span class=&quot;error&quot;&gt;&amp;#91;-59.4, 47.1&amp;#93;&lt;/span&gt;}, \&quot;elevation\&quot;: 9998}]}&quot;})}}&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;&lt;tt&gt;Error:&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;&lt;tt&gt;2022-04-24 22:00:37,917 ERROR Uncaught exception in REST call to /connectors/mongodb-source-connector/topics (org.apache.kafka.connect.runtime.rest.errors.ConnectExceptionMapper) &lt;span class=&quot;error&quot;&gt;&amp;#91;qtp1489322191-15&amp;#93;&lt;/span&gt;&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;javax.ws.rs.NotFoundException: HTTP 404 Not Found&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;&#160; &#160; &#160; &#160; at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:250)&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;&#160; &#160; &#160; &#160; at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;&#160; &#160; &#160; &#160; at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;&#160; &#160; &#160; &#160; at org.glassfish.jersey.internal.Errors.process(Errors.java:292)&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;&#160; &#160; &#160; &#160; at org.glassfish.jersey.internal.Errors.process(Errors.java:274)&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;&#160; &#160; &#160; &#160; at org.glassfish.jersey.internal.Errors.process(Errors.java:244)&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;&#160; &#160; &#160; &#160; at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;&#160; &#160; &#160; &#160; at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;&#160; &#160; &#160; &#160; at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:679)&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&lt;tt&gt;I have verified that the topic &apos;kafka.weather.data&apos; is available but somehow the source connector couldn&apos;t find it with that REST API call.&lt;/tt&gt;&lt;/p&gt;</comment>
                            <comment id="4499211" author="JIRAUSER1269336" created="Fri, 22 Apr 2022 08:50:58 +0000"  >&lt;p&gt;For some reason, the last &apos;pipeline&apos; line of the descriptor file is not showing up properly. Trying again here:&#160;&lt;br/&gt;
[{&quot;$match&quot; :&#160;{&quot;operationType&quot; : &lt;/p&gt;
{ &quot;$in&quot; : [&quot;insert&quot;,&quot;update&quot;,&quot;replace&quot;]}
&lt;p&gt;}}, {&quot;$project&quot;:{&quot;_id&quot;:1,&quot;fullDocument&quot;:1,&quot;ns&quot;:1,&quot;documentKey&quot;:1}}]&lt;/p&gt;</comment>
                            <comment id="4499172" author="JIRAUSER1269336" created="Fri, 22 Apr 2022 08:41:58 +0000"  >&lt;p&gt;Thanks much @Ross Lawley (&lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;mailto:ross@mongodb.com&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;ross@mongodb.com&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/mail_small.gif&quot; height=&quot;12&quot; width=&quot;13&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;). Just realized it&apos;s the source connector that has the issue.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;So I am using the MongoDb source connector to connect to the Kafka topic. Here is the descriptor file:&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;&lt;tt&gt;apiVersion: kafka.strimzi.io/v1beta2&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;kind: KafkaConnector&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;metadata:&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;&#160; name: mongodb-source-connector&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;&#160; labels:&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;&#160; &#160; strimzi.io/cluster: kafka-connect-cluster&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;spec:&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;&#160; class: com.mongodb.kafka.connect.MongoSourceConnector&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;&#160; tasksMax: 2&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;&#160; config:&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;&#160; &#160; connection.uri: &amp;lt;&amp;lt;mongodb_connection_uri&amp;gt;&amp;gt;&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;&#160; &#160; topic.prefix: kafka&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;&#160; &#160; database: weather&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;&#160; &#160; collection: data&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;&#160; &#160; copy.existing: true&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;&#160; &#160; key.converter: org.apache.kafka.connect.json.JsonConverter&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;&#160; &#160; key.converter.schemas.enable: false&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;&#160; &#160; value.converter: org.apache.kafka.connect.json.JsonConverter&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;&#160; &#160; value.converter.schemas.enable: false&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;&#160; &#160; publish.full.document.only: true&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;&#160; &#160; pipeline: &amp;gt;&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;&#160; &#160; &#160; &#160; &#160; &#160; [{&quot;$match&quot; : &lt;/p&gt;
&lt;div class=&quot;error&quot;&gt;&lt;span class=&quot;error&quot;&gt;Unknown macro: { &amp;quot;operationType&amp;quot; }&lt;/span&gt; &lt;/div&gt;
&lt;p&gt;}},{&quot;$project&quot; : {&quot;_id&quot; : 1,&quot;fullDocument&quot; : 1,&quot;ns&quot; : 1,&quot;documentKey&quot; : 1}}]&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;So the first time, all of the Mongodb source collections data is getting copied to the Kafka topic &apos;kafka.weather.data&apos;. But the next time, when I add a new row to my Mongodb source collection, I get the above error.&lt;br/&gt;
&#160;&lt;br/&gt;
Please let me know if you need more information.&#160;&lt;/p&gt;</comment>
                            <comment id="4499136" author="ross@10gen.com" created="Fri, 22 Apr 2022 08:18:10 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=abdul.basith.kj%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;abdul.basith.kj@gmail.com&quot;&gt;abdul.basith.kj@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;The error message is coming from the Source connector and not the sink connector.  Can you provide more information on how you are using the source connector?&lt;/p&gt;

&lt;p&gt;Ross&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                        <issuelink>
            <issuekey id="1370537">KAFKA-111</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i0b354:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                </customfields>
    </item>
</channel>
</rss>