<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:34:39 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-54844] changeStreams supports watch several dbs</title>
                <link>https://jira.mongodb.org/browse/SERVER-54844</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Currently, changeStreams supports watching namespace by 3&#160;granularities: single collection, single db, all dbs. If users want to monitor several dbs, we must watch all dbs and then use the filter stage or some outer method to only pass the namespace we want. However, this method is not efficient because there is so much unwanted data pass between mongos, mongod, and even client if we use the outer method to filter.&lt;/p&gt;

&lt;p&gt;In our multi-tenant severless environment, using &quot;watch all dbs&quot; will consume a lot of resources: db1, db2 belongs to user1; db3, db4 belongs to user2. So when we want to migrate user1 to another instance, we must start changeStreams to watch all dbs(db1, db2).&lt;/p&gt;</description>
                <environment></environment>
        <key id="1635634">SERVER-54844</key>
            <summary>changeStreams supports watch several dbs</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="backlog-query-execution">Backlog - Query Execution</assignee>
                                    <reporter username="cvinllen@gmail.com">vinllen chen</reporter>
                        <labels>
                    </labels>
                <created>Mon, 1 Mar 2021 08:52:18 +0000</created>
                <updated>Sun, 29 Oct 2023 21:56:55 +0000</updated>
                            <resolved>Mon, 11 Oct 2021 14:24:49 +0000</resolved>
                                                    <fixVersion>5.1.0-rc0</fixVersion>
                                    <component>Querying</component>
                                        <votes>1</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="4115735" author="bernard.gorman" created="Mon, 11 Oct 2021 14:24:49 +0000"  >&lt;p&gt;As of release 5.1.0, &lt;tt&gt;$match&lt;/tt&gt; and &lt;tt&gt;$project&lt;/tt&gt; stages will be pushed down from &lt;tt&gt;mongoS&lt;/tt&gt; to the shards where possible. In addition, &lt;tt&gt;$match&lt;/tt&gt; filters will be rewritten such that they can be applied directly to the oplog, for greater efficiency in eliminating non-matching events. It is therefore possible to watch multiple DBs by opening a whole-cluster stream and filtering for something like {&lt;tt&gt;$match: {&quot;ns.db&quot;: {$in: &lt;span class=&quot;error&quot;&gt;&amp;#91;&amp;quot;db1&amp;quot;, &amp;quot;db2&amp;quot;&amp;#93;&lt;/span&gt;&lt;/tt&gt;}}}.&lt;/p&gt;</comment>
                            <comment id="3668188" author="justin.seyster" created="Wed, 17 Mar 2021 03:22:18 +0000"  >&lt;p&gt;Agreed, there is a substantial number of cases to consider for any kind of $match pushdown. For any future pushdown optimization we implement, we&apos;ll try to make it apply to as many scenarios as possible. This kind of feedback is helpful for us to understand which scenarios are useful to address with optimizations, so thanks for providing that additional background!&lt;/p&gt;</comment>
                            <comment id="3668091" author="cvinllen@gmail.com" created="Wed, 17 Mar 2021 02:08:19 +0000"  >&lt;p&gt;Hi, Justin, thanks for your reply.&lt;/p&gt;

&lt;p&gt;Sorry for my previous answer made a mistake, watch all databases will consume a lot of resumes, but watch a single database won&apos;t.&lt;/p&gt;

&lt;p&gt;I agree with your idea that start some separate change stream to each database won&apos;t consume a lot of resources, however, the client needs to do the extra job to guarantee causal consistency between several databases, e.g., If user A has two databases: db1, db2. session1 that watching db1 receives event1(ts=1), event2(ts=10); session2 that watching db2 receives event3(ts=2), event4(ts=11). So the client needs to wait to sort by the timestamp(HLC) across two sessions: ts=1, ts=2, ts=10, ts=11, ...&lt;/p&gt;

&lt;p&gt;Actually, we modify the change stream code to fulfill our goals that watch several databases, we can commit our patch if you need. I think&#160;&lt;a href=&quot;http://example.com/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;SERVER-48694&lt;/a&gt; is a more general request. But in my opinion, it&apos;ll be so many cases to handle if push down all filter into mongod.&#160;I agree this can solve more common problems, looking forward to that.&lt;/p&gt;</comment>
                            <comment id="3667884" author="justin.seyster" created="Tue, 16 Mar 2021 22:32:31 +0000"  >&lt;p&gt;Vinllen, thanks for the detailed description of your use case. You may find that having a separate change stream that is specific to each database is actually more efficient, because even though each change stream will have to read the entire oplog, they will each be able to operate on just the relevant subset of oplog entries, saving significantly on processing time and data transferred from shards to the mongos. It may be worth trying if you haven&apos;t already.&lt;/p&gt;

&lt;p&gt;I agree that being able to push the filter stage into the DocumentSourceOplogMatch would be a useful optimization and would be helpful in this situation. &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-48694&quot; title=&quot;Push down user-defined stages in a change stream pipeline where possible&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-48694&quot;&gt;&lt;del&gt;SERVER-48694&lt;/del&gt;&lt;/a&gt; discusses another use case where such an optimization would be useful, and it includes a helpful comment from &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bernard.gorman&quot; class=&quot;user-hover&quot; rel=&quot;bernard.gorman&quot;&gt;bernard.gorman&lt;/a&gt; outlining some of the details involved in implementing the optimization. We will definitely consider this use case as part of the design for any upcoming change streams optimization work.&lt;/p&gt;</comment>
                            <comment id="3652681" author="JIRAUSER1257066" created="Mon, 8 Mar 2021 18:39:44 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=cvinllen%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;cvinllen@gmail.com&quot;&gt;cvinllen@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for the detailed example. We&apos;re assigning this ticket to the appropriate team to be evaluated against our currently planned work. Updates will be posted on this ticket as they happen.&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Edwin&lt;/p&gt;</comment>
                            <comment id="3650736" author="cvinllen@gmail.com" created="Sat, 6 Mar 2021 10:03:40 +0000"  >&lt;p&gt;Yes, only multiple changeStreams on the database level can fulfill our goals, however, this will consume a lot of resources because every changeStreams needs to fetch all oplog. So I think maybe it&apos;ll be better to support several database granularities or push down the aggregate $filter stage into the DocumentSourceOplogMatch stage, e.g., find all oplog, run the filter, and then run the following stage: DocumentSourceChangeStreamTransform, DocumentSourceCheckInvalidate, DocumentSourceShardCheckResumability, and so on.&lt;/p&gt;</comment>
                            <comment id="3649675" author="JIRAUSER1257066" created="Fri, 5 Mar 2021 17:35:40 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=cvinllen%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;cvinllen@gmail.com&quot;&gt;cvinllen@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for submitting this feature improvement request. Have you found success opening multiple changeStreams on the database level to cover the databases you&apos;d like to watch, rather than watching the entire deployment?&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Edwin&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="1377214">SERVER-48694</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25125"><![CDATA[Query Execution]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 5 Mar 2021 17:35:40 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 17 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-1942</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 17 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-query-execution</customfieldvalue>
            <customfieldvalue>bernard.gorman@mongodb.com</customfieldvalue>
            <customfieldvalue>edwin.zhou@mongodb.com</customfieldvalue>
            <customfieldvalue>justin.seyster@mongodb.com</customfieldvalue>
            <customfieldvalue>cvinllen@gmail.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hyx7hr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr2o7r:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[edwin.zhou@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hywtqv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>