<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 06:53:22 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-83860] Designate a merging shard for nested $lookup stages which target a sharded collection </title>
                <link>https://jira.mongodb.org/browse/SERVER-83860</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;(The scenario below is inspired by &lt;a href=&quot;https://github.com/mongodb/mongo/blame/ff513d9e6b9943737c12e24d81d46ebf074dcef7/jstests/sharding/query/sharded_lookup_execution.js#L385-L409&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;the following test case&lt;/a&gt;, specifically &lt;a href=&quot;https://github.com/mongodb/mongo/blame/ff513d9e6b9943737c12e24d81d46ebf074dcef7/jstests/sharding/query/sharded_lookup_execution.js#L152-L160&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;the following assertion&lt;/a&gt; that we will never push a nested $lookup down to the shards part of the pipeline).&lt;/p&gt;

&lt;p&gt;The targeting behavior of nested $lookup stages was recently changed due to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-79580&quot; title=&quot;Remove HostTypeRequirement::kPrimaryShard from $lookup&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-79580&quot;&gt;&lt;del&gt;SERVER-79580&lt;/del&gt;&lt;/a&gt; (specifically, the &lt;a href=&quot;https://github.com/mongodb/mongo/commit/2a960bb98fc1f05fbb27662af4452845b79294fa#diff-fea6764d9e5346d7a9bfa4d36d2fa83ed942c19840db378a72db2101d6ae544fL493-L496&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;following block&lt;/a&gt; ) Suppose I have a nested $lookup:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;db.a.aggregate([{$lookup: {from: &apos;b&apos;, pipeline: [{$lookup: {from: &apos;c&apos;: &#8230;}], ..}}]_&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;where &apos;a&apos;, &apos;b&apos; and &apos;c&apos; are all sharded. At present, the top level $lookup gets executed in parallel on the shards due to the following logic in &lt;a href=&quot;https://github.com/mongodb/mongo/blob/4ea8dfd60ed7bd3b1ee9568153d4f6a70f361d7e/src/mongo/db/pipeline/document_source_lookup.cpp#L1206-L1215&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;DocumentSourceLookup::distributedPlanLogic()&lt;/a&gt;. This is true before and after &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-79580&quot; title=&quot;Remove HostTypeRequirement::kPrimaryShard from $lookup&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-79580&quot;&gt;&lt;del&gt;SERVER-79580&lt;/del&gt;&lt;/a&gt;. However, the targeting of the inner lookup (that is &lt;br/&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{$lookup:{from: &apos;c&apos;: &#8230;}]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt; against &apos;b&apos; is now different:&lt;/p&gt;

&lt;p&gt;Prior to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-79580&quot; title=&quot;Remove HostTypeRequirement::kPrimaryShard from $lookup&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-79580&quot;&gt;&lt;del&gt;SERVER-79580&lt;/del&gt;&lt;/a&gt;, this nested $lookup would be part of the merging half of the pipeline because the &lt;a href=&quot;https://github.com/mongodb/mongo/commit/2a960bb98fc1f05fbb27662af4452845b79294fa#diff-fea6764d9e5346d7a9bfa4d36d2fa83ed942c19840db378a72db2101d6ae544fL495-L499&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;following check would return kPrimaryShard&lt;/a&gt; (this is because we are not on mongos once the top level $lookup is issued to the shards).&lt;/p&gt;

&lt;p&gt;However, after &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-79580&quot; title=&quot;Remove HostTypeRequirement::kPrimaryShard from $lookup&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-79580&quot;&gt;&lt;del&gt;SERVER-79580&lt;/del&gt;&lt;/a&gt;, this nested $lookup is now pushed to the shards part of the pipeline because we always indicate a host type of &apos;kNone&apos;, &lt;a href=&quot;https://github.com/mongodb/mongo/blob/4ea8dfd60ed7bd3b1ee9568153d4f6a70f361d7e/src/mongo/db/pipeline/sharded_agg_helpers.cpp#L1042-L1045&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;meaning that we no longer have a reason to split the pipeline and put $lookup on the merging half&lt;/a&gt; (previously, the kPrimaryShard host type requirement was enough to do this).&lt;/p&gt;

&lt;p&gt;While this change in targeting behavior is not an issue in terms of correctness, it is arguably a regression (or at least, undesirable) from a performance perspective: it is arguably better to stick the $lookup in the merging half of the pipeline and only run a single one as opposed to pushing down to the shards part of the pipeline and executing one $lookup on each shard that owns a chunk for the collection we are reading from.&lt;/p&gt;

&lt;p&gt;Now, to the point: This ticket proposes to &apos;correct&apos; this behavior by changing DocumentSourceLookUp::constraints to designate the current shard as the merging shard for $lookup when targeting an sharded collection such that we only execute one $lookup in this case.&lt;/p&gt;</description>
                <environment></environment>
        <key id="2515878">SERVER-83860</key>
            <summary>Designate a merging shard for nested $lookup stages which target a sharded collection </summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="mihai.andrei@mongodb.com">Mihai Andrei</assignee>
                                    <reporter username="mihai.andrei@mongodb.com">Mihai Andrei</reporter>
                        <labels>
                    </labels>
                <created>Mon, 4 Dec 2023 19:14:57 +0000</created>
                <updated>Tue, 12 Dec 2023 07:36:12 +0000</updated>
                            <resolved>Mon, 11 Dec 2023 20:26:22 +0000</resolved>
                                                                                        <votes>0</votes>
                                    <watches>1</watches>
                                                                                                                <comments>
                            <comment id="5941923" author="mihai.andrei" created="Mon, 11 Dec 2023 20:26:22 +0000"  >&lt;p&gt;This is addressed in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-79581&quot; title=&quot;Remove HostTypeRequirement::kPrimaryShard from $graphLookup&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-79581&quot;&gt;&lt;del&gt;SERVER-79581&lt;/del&gt;&lt;/a&gt;&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="2406429">SERVER-79581</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="2522449">SERVER-84086</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25125"><![CDATA[Query Execution]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        8 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/SERVER-79581'>SERVER-79581</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-3229</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>dbeng-pm-bot</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            8 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>mihai.andrei@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i3349b:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i2k1sg:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="7261">QE 2023-12-11</customfieldvalue>
    <customfieldvalue id="7264">QE 2023-12-25</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i32qen:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>