<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:51:27 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-39238] Allow foreign $lookup pipeline split at DocumentSourceSequentialDocumentCache</title>
                <link>https://jira.mongodb.org/browse/SERVER-39238</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;As of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-32308&quot; title=&quot;Add the ability for a $lookup stage to execute on mongos against a sharded foreign collection&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-32308&quot;&gt;&lt;del&gt;SERVER-32308&lt;/del&gt;&lt;/a&gt;, it is possible for a $lookup sub-pipeline to be run remotely, either in totality or split with local and remote parts. In the case of a pipeline with a non-correlated pipeline prefix, we will only cache that prefix if the pipeline is split and the cache lives on the local (merging) part of the pipeline.&lt;/p&gt;

&lt;p&gt;We could optimize for the caching of non-correlated pipeline prefix by splitting at the DocumentSourceSequentialDocumentCache stage in cases where we would otherwise not split or would split prior to this stage. There are 2 options for this:&lt;/p&gt;

&lt;p&gt;First, a limited approach that would only split if the DocumentSourceSequentialDocumentCache stage could cache the entire subpipeline, which is uncorrelated by definition. In this case, attempting to cache will not result in additional network traffic, and may reduce traffic if the foreign data size is smaller than the maximum cache size of 100MB.&lt;/p&gt;

&lt;p&gt;The second option, would be a more generalized approach where we always split at the DocumentSourceSequentialDocumentCache stage when there is a non-correlated pipeline prefix that would not otherwise be part of the local pipeline. In this case, there are a few things to consider:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;It is possible that the foreign pipeline has a non-correlated prefix followed by a correlated stage that significantly reduces the size of the result set. In this case we could be sending a large volume of data across shards as compared to running the entire pipeline remotely (for every local document) and sending a small volume of data.&lt;/li&gt;
	&lt;li&gt;The above is mitigated by the fact that we would only retrieve for the first local document and would then use the cache from then on, given we don&apos;t exceed the cache limit. If the 100MB cache limit was exceeded, we could rewrite the pipeline to execute entirely on the remote shard. We could abort any foreign pipeline execution if it reached 100MB (to prevent sending an unbounded amount of data across the wire) and run the entire subpipeline remotely.&lt;/li&gt;
&lt;/ol&gt;
</description>
                <environment></environment>
        <key id="679189">SERVER-39238</key>
            <summary>Allow foreign $lookup pipeline split at DocumentSourceSequentialDocumentCache</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="10038" iconUrl="https://jira.mongodb.org/images/icons/subtask.gif" description="">Backlog</status>
                    <statusCategory id="2" key="new" colorName="default"/>
                                    <resolution id="-1">Unresolved</resolution>
                                        <assignee username="backlog-query-optimization">Backlog - Query Optimization</assignee>
                                    <reporter username="james.wahlin@mongodb.com">James Wahlin</reporter>
                        <labels>
                    </labels>
                <created>Mon, 28 Jan 2019 21:20:55 +0000</created>
                <updated>Tue, 6 Dec 2022 03:07:07 +0000</updated>
                                                                                                <votes>0</votes>
                                    <watches>2</watches>
                                                                                                                    <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="471792">SERVER-32308</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25126"><![CDATA[Query Optimization]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 2 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/SERVER-32308'>SERVER-32308</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>false</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>alexander.golin@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 2 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-query-optimization</customfieldvalue>
            <customfieldvalue>james.wahlin@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hukg5r:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr2glz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|huk2f3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>