<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 09:05:38 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[KAFKA-121] MongoDB Kafka source connector always has single task</title>
                <link>https://jira.mongodb.org/browse/KAFKA-121</link>
                <project id="16285" key="KAFKA">Kafka Connector</project>
                    <description>&lt;p&gt;I am trying to setup MongoDB-Kafka-Connect as Source listening to change-stream of one collection on my windows machine. I am running three worker nodes on localhost windows machine on ports 8083, 8084, 8085.&lt;/p&gt;

&lt;p&gt;I am able to create connector with one task and receive change stream events successfully and fail lover to other worker node is also working fine. But the number of tasks spawned is always one. I have set tasks.max=10, and used multiple threads to generate high volume of change stream events but even then number of task remains one. &lt;/p&gt;

&lt;p&gt;This kind of makes Kafka-producer(MongoDB source connector) not scalable at all. While with multiple partitions on the same topic, consumers are scalable. Is this an issue that I am experiencing or MongoDB-Kafka-Source connector is designed this way?&lt;/p&gt;</description>
                <environment>Windows 10, 8GB RAM, AMD Ryzen 5 2500U, 64Bit, HDD</environment>
        <key id="1406178">KAFKA-121</key>
            <summary>MongoDB Kafka source connector always has single task</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13202">Works as Designed</resolution>
                                        <assignee username="ross@mongodb.com">Ross Lawley</assignee>
                                    <reporter username="jawaid.hamid@gmail.com">Hamid Jawaid</reporter>
                        <labels>
                    </labels>
                <created>Fri, 10 Jul 2020 10:13:20 +0000</created>
                <updated>Fri, 27 Oct 2023 11:54:15 +0000</updated>
                            <resolved>Fri, 10 Jul 2020 13:24:27 +0000</resolved>
                                    <version>1.1</version>
                                                    <component>Source</component>
                                        <votes>0</votes>
                                    <watches>2</watches>
                                                                                                                <comments>
                            <comment id="3281398" author="JIRAUSER1254067" created="Sun, 12 Jul 2020 06:32:38 +0000"  >&lt;p&gt;Thanks Ross.&lt;/p&gt;

&lt;p&gt;Yes, I can have multiple connectors but then I would have to manage duplicates processing of change stream events as both connectors would receive same events. I can provide them different pipeline so that change stream events are not duplicated.&lt;/p&gt;

&lt;p&gt;Thanks for making it clear.&lt;/p&gt;

&lt;p&gt;Documentation: &lt;a href=&quot;https://docs.mongodb.com/kafka-connector/v1.1/kafka-source/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://docs.mongodb.com/kafka-connector/v1.1/kafka-source/&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Can documentation be updated with &quot;tasks.max&quot; not having any influence on number of generated tasks?&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="3279305" author="ross@10gen.com" created="Fri, 10 Jul 2020 13:24:27 +0000"  >&lt;p&gt;The source connector will only ever produce a single task.  &lt;/p&gt;

&lt;p&gt;This is by design as the source connector is backed by a change stream.  Change streams internally use the same data as used by replication engine and as such should be able to scale as the database does.&lt;/p&gt;

&lt;p&gt;There are no plans to allow multiple cursors, however, should you feel that this is not meeting your requirements, then you can configure multiple connectors and each would have its own change stream cursor.&lt;/p&gt;

&lt;p&gt;Ross&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10320">
                    <name>Documented</name>
                                                                <inwardlinks description="is documented by">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10257" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Documentation Changes</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10250"><![CDATA[Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_14266" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Documentation Changes Summary</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;If I set tasks.max=10, using single topic(three partitions), I still see only one task while the fail over to another worker is working fine.&lt;br/&gt;
Spawning(when and how) of source connector &quot;tasks&quot; is not mentioned in documentation of Official MongoDB source connector. Is this the expected behavior? If yes, then documentation should mention that.&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hxh71b:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                </customfields>
    </item>
</channel>
</rss>