<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:12:58 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-46979] Improve changeStream performance relative to oplog queries</title>
                <link>https://jira.mongodb.org/browse/SERVER-46979</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;I&apos;ve made a test between reading oplog and change stream, it looks like the change stream performance is not as good as reading oplog directly. Here comes my experiment:&lt;/p&gt;

&lt;p&gt;The fetcher program locates on hostB while the MongoDB locates on hostA, the ping delay from hostA to hostB is about 0.2~0.3ms. There&apos;re no cpu/memory/io/network bottleneck problem in my experiment.&lt;br/&gt;
I have 500w oplog with total size 5.5G on the source MongoDB: &lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;Change stream&#65306;180 seconds. near 3w qps&lt;/li&gt;
	&lt;li&gt;oplog&#65306;60 seconds. about 8w+ qps&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;In the monitor of change stream fetching, the CPU runs about 60%. I think the gap is slightly bigger. Does this basically match your previous test results?&lt;br/&gt;
As I knew, for a replica set, the change stream will be split into two parts. The first is the $match oplog $cursor that can be seen on the aggregate explain command. The second part is transforming that will do some steps: &lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;unmarshal oplog bson&lt;/li&gt;
	&lt;li&gt;allocate new memory and transform parsed oplog to change stream event&lt;/li&gt;
	&lt;li&gt;marshal change stream event into bson&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;So the main reason for the performance gap is the transforming step. Please let me known if I am wrong.&lt;/p&gt;

&lt;p&gt;In my point of view, there is only 1 thread in oplog fetching and transforming, so increasing the threads can improve the performance. But it will be a tradeoff because it may affect the MongoDB server performance if too many threads are used to do the transform.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1281817">SERVER-46979</key>
            <summary>Improve changeStream performance relative to oplog queries</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13204">Community Answered</resolution>
                                        <assignee username="backlog-server-triage">Backlog - Triage Team</assignee>
                                    <reporter username="cvinllen@gmail.com">vinllen chen</reporter>
                        <labels>
                            <label>change-streams-improvements</label>
                    </labels>
                <created>Thu, 19 Mar 2020 09:43:46 +0000</created>
                <updated>Fri, 27 Oct 2023 15:56:35 +0000</updated>
                            <resolved>Thu, 26 Mar 2020 13:23:10 +0000</resolved>
                                                                    <component>Querying</component>
                                        <votes>0</votes>
                                    <watches>11</watches>
                                                                                                                <comments>
                            <comment id="3658551" author="JIRAUSER1259006" created="Thu, 11 Mar 2021 08:14:14 +0000"  >&lt;p&gt;I&apos;m having this problem as well; I&apos;m listening on a large Mongo cluster, use the&#160;Change stream,&#160;It takes about 20s for 10000 terms, but about 100 million new data are being added every day, this is not acceptable.&lt;/p&gt;</comment>
                            <comment id="3002162" author="dmitry.agranat" created="Thu, 26 Mar 2020 13:22:54 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=cvinllen%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;cvinllen@gmail.com&quot;&gt;cvinllen@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Glad to hear the reply was helpful.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                            <comment id="3002016" author="cvinllen@gmail.com" created="Thu, 26 Mar 2020 12:12:46 +0000"  >&lt;p&gt;Hi Dima,&lt;/p&gt;

&lt;p&gt;Thanks for your reply,  it is helpful. You&apos;re right, I should compare the change stream listening to the newest events to tailing oplog instead of scanning oplog collection.&lt;br/&gt;
And your future plans about increasing the performance sound great, looking forward to that.&lt;/p&gt;</comment>
                            <comment id="2993896" author="dmitry.agranat" created="Tue, 24 Mar 2020 12:05:58 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=cvinllen%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;cvinllen@gmail.com&quot;&gt;cvinllen@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for raising this ticket. Below, I&apos;ve attempted to respond to your comments; please let me know if you have any additional questions.&lt;/p&gt;

&lt;h3&gt;&lt;a name=&quot;Summary&quot;&gt;&lt;/a&gt;Summary&lt;/h3&gt;

&lt;p&gt;It is true that &lt;tt&gt;$changeStream&apos;s&lt;/tt&gt; raw read rate is (unavoidably) slower than a simple query on the oplog, that the &lt;tt&gt;$changeStream&apos;s&lt;/tt&gt; oplog &lt;tt&gt;$match&lt;/tt&gt; and transformation stages contribute most to this latency, and that CPU consumption is higher when running these stages as compared to a basic query. However, in our tests of oplog-tailing versus &lt;tt&gt;$changeStream&lt;/tt&gt; under typical operating conditions and write workloads, performance of each has been similar up to the point where resource contention becomes an issue.&lt;/p&gt;

&lt;p&gt;From the description of your experiment, it sounds as though you were not tailing the oplog, but instead started a change stream at the beginning of the oplog and compared it against a straightforward collection scan. The most important point to make is that this is &lt;b&gt;not the use-case for which change streams are designed.&lt;/b&gt; Change streams are not intended to produce the same raw read rate as a basic query for a one-off scan through the entire contents of a large oplog; they are intended to operate towards the end of the oplog, scanning the entries added within the past few seconds, minutes, or hours. While it is sometimes unavoidable to scan through the oplog, we actively attempt to &lt;b&gt;avoid&lt;/b&gt; doing so more than is strictly necessary; for instance, we implemented the &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-38412&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;Change Streams High Water Mark&lt;/a&gt; project specifically to avoid having to do this when resuming a stream.&lt;/p&gt;

&lt;p&gt;We will continue to work on improving &lt;tt&gt;$changeStream&lt;/tt&gt;, but the fact that there is a difference in performance between these use-cases is expected. To better reflect the issue you raised, we&apos;ve changed the title of this ticket to &quot;Improve &lt;tt&gt;$changeStream&lt;/tt&gt; performance relative to oplog queries.&quot;&lt;/p&gt;

&lt;h3&gt;&lt;a name=&quot;Regardingyourcommentsandquestions&quot;&gt;&lt;/a&gt;Regarding your comments and questions&lt;/h3&gt;

&lt;blockquote&gt;&lt;p&gt;As I knew, for a replica set, the change stream will be split into two parts. The first is the $match oplog $cursor that can be seen on the aggregate explain command. The second part is transforming that will do some steps:&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;This is broadly accurate, and for the purposes of this conversation these are certainly the heaviest stages in a simple change stream pipeline.&lt;/p&gt;

&lt;blockquote&gt;&lt;p&gt;In the monitor of change stream fetching, the CPU runs about 60%. I think the gap is slightly bigger. Does this basically match your previous test results? So the main reason for the performance gap is the transforming step. Please let me known if I am wrong.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;Based on our internal tests, change stream CPU consumption is indeed notably higher than a simple oplog query, both because of the complex &lt;tt&gt;$match&lt;/tt&gt; filter that change streams must apply to the oplog, and especially because of the &lt;tt&gt;DocumentSourceChangeStreamTransform&lt;/tt&gt; stage. The latter is responsible for most of the additional execution latency.&lt;/p&gt;

&lt;blockquote&gt;&lt;p&gt;In my point of view, there is only 1 thread in oplog fetching and transforming, so increasing the threads can improve the performance. But it will be a tradeoff because it may affect the MongoDB server performance if too many threads are used to do the transform.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;Parallel query/aggregation execution is indeed on our longer-term roadmap, but in the meantime, we would probably not consider making individual stages like &lt;tt&gt;DocumentSourceChangeStreamTransform&lt;/tt&gt; internally multithreaded. Given that the &lt;tt&gt;$match&lt;/tt&gt; and transformation stages are often very similar across change streams, we have also discussed the possibility of sharing the execution machinery between them where possible; but this is more about collapsing the work of multiple separate streams into a single source rather than multithreading each individual stream.&lt;/p&gt;

&lt;p&gt;I hope the above information is useful, and many thanks again for bringing this issue to our attention!&lt;/p&gt;

&lt;p&gt;Best regards,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25137"><![CDATA[Server Triage]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 19 Mar 2020 11:54:32 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 47 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-1943</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 47 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-server-triage</customfieldvalue>
            <customfieldvalue>dmitry.agranat@mongodb.com</customfieldvalue>
            <customfieldvalue>cvinllen@gmail.com</customfieldvalue>
            <customfieldvalue>lianpengcheng2016@gmail.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hx9st3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hwxjyf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hx9f2f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>