<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:23:10 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-30252] Write oplog operations to kafka</title>
                <link>https://jira.mongodb.org/browse/SERVER-30252</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hi&lt;/p&gt;

&lt;p&gt;When insert/update/delete operation is performed, write it not only to the oplog, but also to a kafka endpoint. The shard name and the collection could be used to determine the kafka partition number.&lt;/p&gt;

&lt;p&gt;Benefits/Use cases:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Easily create sharded clusters by a different shard key&lt;/li&gt;
	&lt;li&gt;Monitoring &amp;amp; auditing of data&lt;/li&gt;
	&lt;li&gt;Apply stream processing on near real time of the data.&lt;/li&gt;
	&lt;li&gt;Provide more consistency in the CAP (see replication).&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Despite this could be implemented via external scripts, they become complicated due to the fact that would need to  also interact with the config servers (or mongos commands, not sure if all the info is available). To detect new nodes in the cluster, also scaling the script/monitor software can become a difficult task while implemented on the mongod process seems simple.&lt;br/&gt;
Questions would be what to do if the Kafka cluster is unavailable, maybe configurable behaviour choice would be good. Despite adding a dependency with an external system as Kafka I think it adds significant benefits. &lt;/p&gt;

&lt;p&gt;Potentially it even could be used to replicate the DBs instead of the oplog, and in the event of the failure of the primary, secondaries would be configured to process the pending oplog messages in kafka before actually marking the mongod instance as online.As kafka is replicated and HA,if the operation made it in the primary we can be sure it can make it to the secondaries. At the end, kafka is a write ahead log, same as oplog. &lt;/p&gt;

&lt;p&gt;JL&lt;/p&gt;

</description>
                <environment></environment>
        <key id="406712">SERVER-30252</key>
            <summary>Write oplog operations to kafka</summary>
                <type id="2" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14711&amp;avatarType=issuetype">New Feature</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="-1">Unassigned</assignee>
                                    <reporter username="jlpedrosa@gmail.com">Jose Luis Pedrosa</reporter>
                        <labels>
                    </labels>
                <created>Fri, 21 Jul 2017 07:57:14 +0000</created>
                <updated>Fri, 21 Jul 2017 19:43:17 +0000</updated>
                            <resolved>Fri, 21 Jul 2017 19:42:27 +0000</resolved>
                                                                                        <votes>0</votes>
                                    <watches>8</watches>
                                                                                                                <comments>
                            <comment id="1629047" author="ramon.fernandez" created="Fri, 21 Jul 2017 19:42:27 +0000"  >&lt;p&gt;There are previous tickets that would enable this:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-13932&quot; title=&quot;Change Notification Stream API &quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-13932&quot;&gt;&lt;del&gt;SERVER-13932&lt;/del&gt;&lt;/a&gt;&lt;/li&gt;
	&lt;li&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-5042&quot; title=&quot;Implement support for reliable change notifications&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-5042&quot;&gt;&lt;del&gt;SERVER-5042&lt;/del&gt;&lt;/a&gt;&lt;/li&gt;
	&lt;li&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-124&quot; title=&quot;triggers &quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-124&quot;&gt;SERVER-124&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Arguably the MongoDB server should not be in charge of feeding other systems, but just enabling other tools to interface with it in a way that permits doing something like what you describe.&lt;/p&gt;

&lt;p&gt;I&apos;m therefore going to close this request as a duplicate. I&apos;d encourage you to watch and vote for the tickets above, so your particular use case can be taken into consideration during the design phase.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="10249">SERVER-124</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="31260">SERVER-5042</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="136339">SERVER-13932</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 21 Jul 2017 17:07:41 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        6 years, 29 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            6 years, 29 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>jlpedrosa@gmail.com</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htbd6v:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|ht37ov:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htaz9j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>