<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 09:05:31 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[KAFKA-76] Reuse the postBatchResumeToken</title>
                <link>https://jira.mongodb.org/browse/KAFKA-76</link>
                <project id="16285" key="KAFKA">Kafka Connector</project>
                    <description>&lt;p&gt;The latest Kafka connector uses Java 3.11 and thus takes advantage of  &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-35740&quot; title=&quot;Report high-water-mark resume token with each (possibly empty) change stream batch&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-35740&quot;&gt;&lt;del&gt;SERVER-35740&lt;/del&gt;&lt;/a&gt; (HighWaterMarkToken).  Under the hood though, it is the resumeToken (offset) and the corresponding event (matching the change stream filter) that are published as topics. If a connector crash, the offset from the last published Source Record is used as the resumeToken upon restart.&lt;/p&gt;

&lt;p&gt;In certain situation, such as when the connector is listening a dormant db/collection, there is a potential for the resumeToken to be out of the oplog upon restart. Saving the postBatchResumeToken will reduce the likelihood of such a failure to occur&lt;/p&gt;</description>
                <environment></environment>
        <key id="1036772">KAFKA-76</key>
            <summary>Reuse the postBatchResumeToken</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="ross@mongodb.com">Ross Lawley</assignee>
                                    <reporter username="davenson.lombard@mongodb.com">Davenson Lombard</reporter>
                        <labels>
                    </labels>
                <created>Wed, 4 Dec 2019 19:11:22 +0000</created>
                <updated>Sat, 28 Oct 2023 10:46:24 +0000</updated>
                            <resolved>Tue, 15 Sep 2020 09:51:44 +0000</resolved>
                                                    <fixVersion>1.3.0</fixVersion>
                                    <component>Source</component>
                                        <votes>3</votes>
                                    <watches>12</watches>
                                                                                                                <comments>
                            <comment id="3394033" author="xgen-internal-githook" created="Tue, 15 Sep 2020 09:46:37 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Ross Lawley&apos;, &apos;email&apos;: &apos;ross.lawley@gmail.com&apos;, &apos;username&apos;: &apos;rozza&apos;}
&lt;p&gt;Message: Added PostBatchResumeToken Support&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/KAFKA-76&quot; title=&quot;Reuse the postBatchResumeToken&quot; class=&quot;issue-link&quot; data-issue-key=&quot;KAFKA-76&quot;&gt;&lt;del&gt;KAFKA-76&lt;/del&gt;&lt;/a&gt;&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo-kafka/commit/f5b15a93851d36ff2ca80a2ca642231ac3121c31&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo-kafka/commit/f5b15a93851d36ff2ca80a2ca642231ac3121c31&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3373979" author="ross@10gen.com" created="Thu, 3 Sep 2020 11:12:22 +0000"  >&lt;p&gt;PR: &lt;a href=&quot;https://github.com/mongodb/mongo-kafka/pull/32&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo-kafka/pull/32&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2585502" author="ross@10gen.com" created="Thu, 5 Dec 2019 10:13:50 +0000"  >&lt;p&gt;The issue is the Kafka Connector API doesn&apos;t support saving offsets without data - the only data that can be stored is with the SourceRecord that is scheduled to be published onto a topic.  As there is no data from the change stream when polling there is nothing that can be published. &lt;/p&gt;

&lt;p&gt;So we can&apos;t store the postBatchResumeToken when polling the cursor if there is no data and there is nothing I&apos;m aware of in the Kafka Connect API that would allow the support for just saving offsets without data.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10320">
                    <name>Documented</name>
                                                                <inwardlinks description="is documented by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="1309483">KAFKA-96</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1297132">KAFKA-93</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="1553070">KAFKA-176</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                        <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[5002K00000huPb7QAE, 5002K00000nmQQeQAM, 5002K00000nn7eEQAQ, 5002K00000nn7clQAA, 5002K00000oegDeQAI, 5002K00000pDcOIQA0, 5002K00000pEUpNQAW, 5002K00000pkt9sQAA]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10257" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Documentation Changes</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10250"><![CDATA[Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_14266" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Documentation Changes Summary</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;Added two new configurations:&lt;/p&gt;

&lt;p&gt;heartbeat.interval.ms=0&lt;/p&gt;

&lt;p&gt;The length of time between sending heartbeat messages to record the post batch resume token  when no source records have been published. Improves the resumability of the connector for low volume namespaces. Use 0 to disable.&lt;/p&gt;

&lt;p&gt;heartbeat.topic.name=&quot;__mongodb_heartbeats&quot;&lt;br/&gt;
The name of the topic to publish heartbeats to. Defaults to &apos;__mongodb_heartbeats&apos;.&lt;/p&gt;

&lt;p&gt;Note:  By default this feature is off, setting the `heartbeat.interval.ms` will turn it on.  If there have been no messages in the `heartbeat.interval.ms` time then the post batch resume token is sent to the heartbeat topic.  Messages on the heartbeat topic have to be consumed so that the latest offset (post batch resume token) is available.&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hw338n:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                </customfields>
    </item>
</channel>
</rss>