<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:43:14 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-36482] Add mapping of namespace to ChunkVersion for versioned commands involving multiple collections</title>
                <link>https://jira.mongodb.org/browse/SERVER-36482</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;This becomes very relevant for $out to a sharded collection as well as sharded $lookup.  Currently the cluster aggregate command will attach a single version based on the state of the execution namespace, however this may be incorrect if one of the shards wishes grab a collection lock on one of the other involved namespaces.  &lt;/p&gt;

&lt;p&gt;The title describes one possible approach, though I&apos;m not sure how the implementation would look since it may involve modifying the operation context. Another option would be to have the shards temporarily modify the version on the op context when accessing a foreign collection.&lt;/p&gt;

&lt;p&gt;Edit: Looks like such a map exists already, so the problem is that the involved namespaces do not have entries and thus default to unsharded.&lt;/p&gt;</description>
                <environment></environment>
        <key id="584970">SERVER-36482</key>
            <summary>Add mapping of namespace to ChunkVersion for versioned commands involving multiple collections</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="2">Won&apos;t Fix</resolution>
                                        <assignee username="kyle.suarez@mongodb.com">Kyle Suarez</assignee>
                                    <reporter username="nicholas.zolnierz@mongodb.com">Nicholas Zolnierz</reporter>
                        <labels>
                    </labels>
                <created>Tue, 7 Aug 2018 15:51:02 +0000</created>
                <updated>Tue, 28 Aug 2018 20:27:19 +0000</updated>
                            <resolved>Tue, 28 Aug 2018 20:27:19 +0000</resolved>
                                                                    <component>Aggregation Framework</component>
                    <component>Sharding</component>
                                        <votes>0</votes>
                                    <watches>8</watches>
                                                                                                                <comments>
                            <comment id="1988148" author="kyle.suarez" created="Tue, 28 Aug 2018 20:27:19 +0000"  >&lt;p&gt;We&apos;re not going to do this. Our consensus is that a sharded aggregation that touches multiple namespaces will do shard versioning individually for each involved namespace. The infrastructure in the $out project will be tolerant to shard version changes such that writes will always be correctly routed to the shards that own them.&lt;/p&gt;

&lt;p&gt;A related aside from our discussion is that it&apos;s important that sharded aggregations clean up cursors after failure; that work will be tracked in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-36911&quot; title=&quot;Ensure that cluster aggregations that fail clean up cursors&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-36911&quot;&gt;&lt;del&gt;SERVER-36911&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="1974048" author="kyle.suarez" created="Mon, 13 Aug 2018 15:01:23 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kaloian.manassiev&quot; class=&quot;user-hover&quot; rel=&quot;kaloian.manassiev&quot;&gt;kaloian.manassiev&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=esha.maharishi&quot; class=&quot;user-hover&quot; rel=&quot;esha.maharishi&quot;&gt;esha.maharishi&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=nicholas.zolnierz&quot; class=&quot;user-hover&quot; rel=&quot;nicholas.zolnierz&quot;&gt;nicholas.zolnierz&lt;/a&gt;, I put something on the calendar so we can discuss in person, and then I&apos;ll summarize the results of our meeting here.&lt;/p&gt;</comment>
                            <comment id="1973214" author="kaloian.manassiev" created="Sat, 11 Aug 2018 13:27:58 +0000"  >&lt;p&gt;Having multiple collection shardVersions being sent implies some kind of atomicity is required, as in &quot;I am expecting these two collections on this shard to be exactly at these versions&quot;. Since distributed &lt;tt&gt;$lookup&lt;/tt&gt; is in a sense a two-phase operation - first check the main collection&apos;s version and then check the looked-up collection&apos;s version, this doesn&apos;t look like an atomic operation to me.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kyle.suarez&quot; class=&quot;user-hover&quot; rel=&quot;kyle.suarez&quot;&gt;kyle.suarez&lt;/a&gt;, can you schedule a meeting next week to go over the scenarios for &lt;tt&gt;$lookup&lt;/tt&gt; and see whether multiple versions can be avoided? I would also like to go over the usages of &lt;tt&gt;isSharded()&lt;/tt&gt; that &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=nicholas.zolnierz&quot; class=&quot;user-hover&quot; rel=&quot;nicholas.zolnierz&quot;&gt;nicholas.zolnierz&lt;/a&gt; pointed out in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-36482?focusedCommentId=1970690&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-1970690&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; comment.&lt;/p&gt;</comment>
                            <comment id="1970843" author="esha.maharishi@10gen.com" created="Wed, 8 Aug 2018 20:26:04 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kyle.suarez&quot; class=&quot;user-hover&quot; rel=&quot;kyle.suarez&quot;&gt;kyle.suarez&lt;/a&gt;, the shard definitely might not have a fresh routing table for &quot;bar&quot;, in much the same way a mongos might not have a fresh routing table when routing.&lt;/p&gt;

&lt;p&gt;The shard should attach the shardVersion from its CatalogCache when sending remote requests on &quot;bar&quot;, so that the shard can be alerted that its routing table for &quot;bar&quot; was stale and refresh it.&lt;/p&gt;</comment>
                            <comment id="1970786" author="kyle.suarez" created="Wed, 8 Aug 2018 19:49:02 +0000"  >&lt;blockquote&gt;
&lt;p&gt;However, is it really important that the sender believes the shardVersion for &quot;bar&quot; is 6? Or is it simply important that &quot;bar&quot; is allowed to be sharded?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;I believe there&apos;s two things we want: it&apos;s important that &quot;bar&quot; is allowed to be sharded &lt;b&gt;and&lt;/b&gt; that we know that we&apos;re going to have an up-to-date routing table with regard to which shards own which chunks for &quot;bar&quot;. &lt;/p&gt;</comment>
                            <comment id="1970748" author="esha.maharishi@10gen.com" created="Wed, 8 Aug 2018 19:26:13 +0000"  >&lt;p&gt;Yeah. Here is my brain dump for when you all triage this ticket:&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;shardVersion can be sent for two reasons:&lt;/p&gt;

&lt;p&gt;1) To ensure the sender targeted the right set of shards&lt;/p&gt;

&lt;p&gt;2) To ensure a collection was unsharded, as the sender believed.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Today, a shardVersion sent in a network message looks like this:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;shardVersion: 5&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;This is interpreted by the shard as:&lt;/p&gt;

&lt;p&gt;1) the sender&apos;s believes shardVersion for the &quot;main&quot; namespace for the command (e.g., the value of the first key in the cmdObj) is 5.&lt;/p&gt;

&lt;p&gt;2) the sender expects &lt;em&gt;all other namespaces accessed by the command&lt;/em&gt; to be unsharded.&lt;/p&gt;

&lt;p&gt;Until 4.0, since both $out and $lookup always took CollectionLocks that did shardVersion checks, both $out and $lookup were guaranteed to only run on unsharded collections.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;I think this ticket is requesting mongos to be able to send something like:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;shardVersions: {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    &quot;foo&quot;: 5,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    &quot;bar&quot;: 6&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;} &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;and for the shard to interpret this message to mean:&lt;/p&gt;

&lt;p&gt;1) the sender believes the shardVersion for &quot;foo&quot; is 5 and for &quot;bar&quot; is 6&lt;/p&gt;

&lt;p&gt;2) the sender expects&#160;&lt;em&gt;all other namespaces accessed besides &quot;foo&quot; and &quot;bar&quot;&lt;/em&gt; to be unsharded.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;However, is it really important that the sender believes the shardVersion for &quot;bar&quot; is 6? Or is it simply important that &quot;bar&quot; is allowed to be sharded?&lt;/p&gt;

&lt;p&gt;If the latter, the shard can access &quot;bar&quot; under a collection lock that does not do a shardVersion check.&lt;/p&gt;</comment>
                            <comment id="1970690" author="nicholas.zolnierz" created="Wed, 8 Aug 2018 18:48:27 +0000"  >&lt;p&gt;Yeah there&apos;s likely no problem for $out at the moment, but it still feels fragile in places where we use the auto getter which checks the shard version such as  &lt;a href=&quot;https://github.com/mongodb/mongo/blob/master/src/mongo/db/pipeline/mongod_process_interface.cpp#L139&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://github.com/mongodb/mongo/blob/master/src/mongo/db/pipeline/mongod_process_interface.cpp#L268&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;. I know the first spot is a bug which I have fixed in a follow-up, but I&apos;m less certain about the second which is in a method used by $lookup. &lt;/p&gt;</comment>
                            <comment id="1970593" author="kyle.suarez" created="Wed, 8 Aug 2018 17:41:22 +0000"  >&lt;p&gt;I agree with the assessment that it&apos;s not a strict requirement for $out. I do think that it will be a problem for $lookup, though, because in a sharded $lookup world there will be any number of sharded or unsharded (&quot;single chunk&quot;) collections in the shards part of the pipeline.&lt;/p&gt;</comment>
                            <comment id="1970582" author="esha.maharishi@10gen.com" created="Wed, 8 Aug 2018 17:35:47 +0000"  >&lt;p&gt;Hmm. Maybe someone from sharding can work with query to figure out what the exact requirement is. I think it&apos;s possible that this is not required for the $out project (I am not sure about $lookup), because for $out:&lt;/p&gt;

&lt;p&gt;1) mongos targets shards based on the input collection&apos;s chunk distribution, so it only needs to send the shardVersion for the input collection&lt;/p&gt;

&lt;p&gt;2) the shard targets shards based on the output collection&apos;s chunk distribution, so it only needs to send shardVersion for the output collection&lt;/p&gt;

&lt;p&gt;Therefore, no node needs to send shardVersion for two namespaces.&lt;/p&gt;</comment>
                            <comment id="1970479" author="kyle.suarez" created="Wed, 8 Aug 2018 16:28:50 +0000"  >&lt;p&gt;CC &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kaloian.manassiev&quot; class=&quot;user-hover&quot; rel=&quot;kaloian.manassiev&quot;&gt;kaloian.manassiev&lt;/a&gt;&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="596202">SERVER-36911</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>10.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 8 Aug 2018 16:28:50 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 24 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>kyle.suarez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 24 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>esha.maharishi@mongodb.com</customfieldvalue>
            <customfieldvalue>kaloian.manassiev@mongodb.com</customfieldvalue>
            <customfieldvalue>kyle.suarez@mongodb.com</customfieldvalue>
            <customfieldvalue>nicholas.zolnierz@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hu4xcf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr8j9b:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2466">Query 2018-09-10</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hu4jlr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>