<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:42:01 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-57503] Investigate up-front view resolution for $lookup on sharded views</title>
                <link>https://jira.mongodb.org/browse/SERVER-57503</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;The current design for $lookup support for sharded views is to perform view resolution at query runtime, which is not ideal from a design standpoint and could result in suboptimal performance in the case of nested $lookups. Investigate the engineering effort of resolving the view up front instead.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1770858">SERVER-57503</key>
            <summary>Investigate up-front view resolution for $lookup on sharded views</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="alya.berciu@mongodb.com">Alya Berciu</assignee>
                                    <reporter username="katherine.wu@mongodb.com">Katherine Wu</reporter>
                        <labels>
                            <label>read-only-views</label>
                    </labels>
                <created>Mon, 7 Jun 2021 17:51:44 +0000</created>
                <updated>Tue, 6 Jul 2021 21:26:26 +0000</updated>
                            <resolved>Tue, 22 Jun 2021 16:16:15 +0000</resolved>
                                                                                        <votes>0</votes>
                                    <watches>2</watches>
                                                                                                                <comments>
                            <comment id="3919250" author="katherine.wu" created="Tue, 6 Jul 2021 21:26:26 +0000"  >&lt;p&gt;A slight addition to &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=alya.berciu&quot; class=&quot;user-hover&quot; rel=&quot;alya.berciu&quot;&gt;alya.berciu&lt;/a&gt;&apos;s comment above: if there was a nested $lookup with the &apos;from&apos; field as a sharded view, doing the view resolution after targeting will result in slightly worse performance, since the top-level $lookup will rebuild the subpipeline for every input document, and so the nested $lookup will have to handle the CommandOnShardedViewNotSupportedOnMongod once every time the top-level $lookup subpipeline is run.&lt;/p&gt;

&lt;p&gt;If we wanted to circumvent this we&apos;d have to either traverse through the pipeline and subpipelines of every stage to look for a $lookup and resolve the &apos;from&apos; namespace (which has its difficulties as Alya mentioned above), or edit the stages inside the top-level DocumentSourceLookup&apos;s `_resolvedPipeline` the first time a nested $lookup encounters a CommandOnShardedViewNotSupportedOnMongod error, which seems hacky and prone to other problems. Given that $unionWith is already handling a similar situation with the CommandOnShardedViewNotSupportedOnMongod approach, it seems like we should also follow this approach for the sharded $lookup case.&lt;/p&gt;</comment>
                            <comment id="3891772" author="alya.berciu" created="Tue, 22 Jun 2021 16:16:15 +0000"  >&lt;p&gt;I took a look at whether it would be possible to do the view resolution before splitting the pipelines as &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=katherine.wu&quot; class=&quot;user-hover&quot; rel=&quot;katherine.wu&quot;&gt;katherine.wu&lt;/a&gt;&#160;suggested. I don&apos;t think we should try to do this as part of the sharded lookup project.&lt;/p&gt;

&lt;p&gt;First, it is worth noting that $unionWith already handles this as described in the design document, where we just handle a&#160;&lt;br/&gt;
 CommandOnShardedViewNotSupportedOnMongod exception in situations where we need to obtain a sharded view definition, and then reconstruct a pipeline based on the view definition enclosed in the exception itself:&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/blob/3befdc7d70fa56085bbdc9606da0db84b5b48ccd/src/mongo/db/pipeline/document_source_union_with.cpp#L194-L206&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/3befdc7d70fa56085bbdc9606da0db84b5b48ccd/src/mongo/db/pipeline/document_source_union_with.cpp#L194-L206&lt;/a&gt;&lt;/p&gt;


&lt;p&gt; The problem that I ran into was that we don&apos;t currently have a good way to resolve views using the primary shard&apos;s view catalog directly in mongos.&lt;/p&gt;

&lt;p&gt;Since we are not on a mongod, we can&apos;t access the primary&apos;s view catalog to resolve any views. We also can&apos;t generate the&#160;CommandOnShardedViewNotSupportedOnMongod exception before actually targeting the shards (the primary will always know the view definition). Doing this after targeting would result in the same overhead as handling the exception in the mongod of each shard that executes its portion of the pipeline.&lt;/p&gt;

&lt;p&gt;We would need to implement a command (or something similar) to request the view definition from the primary shard&apos;s view catalog so we can update the pipeline with the view definitions before splitting it. However, as Katherine pointed out, this would be at best a temporary measure until global view catalogs get implemented.&lt;/p&gt;</comment>
                            <comment id="3863193" author="katherine.wu" created="Mon, 7 Jun 2021 18:30:47 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kyle.suarez&quot; class=&quot;user-hover&quot; rel=&quot;kyle.suarez&quot;&gt;kyle.suarez&lt;/a&gt;&#160;the initial design for this was to handle the view resolution during the query&apos;s execution in the DocumentSourceLookUp&apos;s doGetNext() function; an &quot;up front&quot; alternative might be to resolve the view on mongos by querying the view catalog stored on the db&apos;s primary shard and updating the DocumentSourceLookUp accordingly, before sending it to be executed on the shards.&lt;/p&gt;</comment>
                            <comment id="3863127" author="kyle.suarez" created="Mon, 7 Jun 2021 18:12:11 +0000"  >&lt;p&gt;What do we mean by &quot;up front&quot; here if it&apos;s not at query runtime? Would it be at the time of view creation (or any catalog change)?&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="478952">SERVER-32548</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 7 Jun 2021 18:12:11 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 31 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-282</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>katherine.wu@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 31 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>alya.berciu@mongodb.com</customfieldvalue>
            <customfieldvalue>katherine.wu@mongodb.com</customfieldvalue>
            <customfieldvalue>kyle.suarez@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hzk37j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr369r:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="4708">Query Optimization 2021-06-28</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hzjpgn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>