<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:44:10 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-36813] Test the defaulting of uniqueKey when the catalog cache is stale</title>
                <link>https://jira.mongodb.org/browse/SERVER-36813</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We added some testing of what happens when an $out is executing with a stale cache, but we&apos;d like to target the functionality introduced in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-35954&quot; title=&quot;Build uniqueKey from the shard key if not present in user command&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-35954&quot;&gt;&lt;del&gt;SERVER-35954&lt;/del&gt;&lt;/a&gt; which will also read the cache.&lt;/p&gt;

&lt;p&gt;This will include the interesting test case of seeing what unique key is used if a mongos has a stale version of the cache which indicates that a collection is sharded but it actually is not, or if it is sharded but now with a different key.&lt;/p&gt;</description>
                <environment></environment>
        <key id="593060">SERVER-36813</key>
            <summary>Test the defaulting of uniqueKey when the catalog cache is stale</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="charlie.swanson@mongodb.com">Charlie Swanson</assignee>
                                    <reporter username="charlie.swanson@mongodb.com">Charlie Swanson</reporter>
                        <labels>
                    </labels>
                <created>Wed, 22 Aug 2018 22:36:51 +0000</created>
                <updated>Sun, 29 Oct 2023 22:28:43 +0000</updated>
                            <resolved>Thu, 1 Nov 2018 21:02:02 +0000</resolved>
                                                    <fixVersion>4.1.5</fixVersion>
                                    <component>Aggregation Framework</component>
                    <component>Sharding</component>
                                        <votes>0</votes>
                                    <watches>5</watches>
                                                                                                                <comments>
                            <comment id="2049482" author="xgen-internal-githook" created="Thu, 1 Nov 2018 21:00:57 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Charlie Swanson&apos;, &apos;email&apos;: &apos;charlie.swanson@mongodb.com&apos;, &apos;username&apos;: &apos;cswanson310&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-36813&quot; title=&quot;Test the defaulting of uniqueKey when the catalog cache is stale&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-36813&quot;&gt;&lt;del&gt;SERVER-36813&lt;/del&gt;&lt;/a&gt; Be careful when choosing default uniqueKey&lt;/p&gt;

&lt;p&gt;Before doing so, refresh the catalog cache to make sure the mongos&lt;br/&gt;
serving the request is at least somewhat up to date. Additionally,&lt;br/&gt;
communicate the epoch used to choose the uniqueKey from mongos to the&lt;br/&gt;
shards, and raise an error from the shards and terminate the&lt;br/&gt;
aggregation if the epoch of the targeted collection ever changes.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/9282c05723eb9f15a6591613007ebe68561c88cb&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/9282c05723eb9f15a6591613007ebe68561c88cb&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1992750" author="charlie.swanson" created="Tue, 4 Sep 2018 17:35:23 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kaloian.manassiev&quot; class=&quot;user-hover&quot; rel=&quot;kaloian.manassiev&quot;&gt;kaloian.manassiev&lt;/a&gt; failing the cluster writes will not fully solve the problem. At that point we have already generated a whole batch of writes with the wrong shard key, which may have led to errors like those described above relating to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-36367&quot; title=&quot;Add validation checks for $out uniqueKey&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-36367&quot;&gt;&lt;del&gt;SERVER-36367&lt;/del&gt;&lt;/a&gt;. It seems like a full solution would either be gross and complicated and be able to store up errors like that and retry a batch on a stale epoch/changed uuid, &lt;em&gt;or&lt;/em&gt; it would involve some sort of round trip up-front to make sure your cache of the shard key is up to date.&lt;/p&gt;</comment>
                            <comment id="1992744" author="kaloian.manassiev" created="Tue, 4 Sep 2018 17:31:42 +0000"  >&lt;p&gt;Failing cluster writes on epoch mismatch will solve the problem with the shard key changing from underneath a long executing $out aggregation, so the &lt;tt&gt;listIndexes&lt;/tt&gt; call is only an optimization to avoid restarting a long-running aggregation on the first stale version, right?&lt;/p&gt;

&lt;p&gt;Otherwise, what kind of bugs is there a possibility for?&lt;/p&gt;</comment>
                            <comment id="1992734" author="nicholas.zolnierz" created="Tue, 4 Sep 2018 17:26:33 +0000"  >&lt;p&gt;It&apos;s unfortunate, but sounds like a good plan to me. Per in-person discussion, we may take a perf hit in latency but its unlikely to affect users since the $out will likely be bottlenecked in the writes. &lt;/p&gt;</comment>
                            <comment id="1992638" author="charlie.swanson" created="Tue, 4 Sep 2018 16:32:27 +0000"  >&lt;p&gt;Ok I&apos;ve brainstormed a lot and I&apos;ll post a brief update here. There is definitely a large potential for bugs here, because we are not using any sort of shard versioning protocol on the $out namespace, only the source namespace of the aggregation. For example, if your mongos is very stale - either because no one has asked it about the $out collection in a while, or because it was just restarted, the mongos which parses the $out could end up filling in the wrong uniqueKey for the $out stage. Or, imagine the aggregation takes a couple hours to run, if the collection becomes sharded or is dropped and recreated during that period, the uniqueKey could be stale.&lt;/p&gt;

&lt;p&gt;The user-impact of using the wrong default uniqueKey is hard to quantify. Because the default uniqueKey will always include the _id, and because you can only end up with duplicate _ids if you managed to insert them on different shards, there will likely still be at most one document matching the uniqueKey for each document output from the $out stage - so pretty low likelihood in a difference of behavior there. However, we will enforce certain constraints, as described in&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-36367&quot; title=&quot;Add validation checks for $out uniqueKey&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-36367&quot;&gt;&lt;del&gt;SERVER-36367&lt;/del&gt;&lt;/a&gt;. This could be weird if we&apos;re using the wrong uniqueKey.&lt;/p&gt;

&lt;p&gt;While investigating this issue, I realized that the targeter used inside the ClusterWriter for all writes on mongos will keep retrying the writes, even if the collection has been dropped and re-sharded - or has become sharded recently. This is similar in the way that it probably will usually do the right thing, but doesn&apos;t feel obviously correct. For example, imagine you are halfway through applying a batch of writes and the collection is dropped and re-created. It&apos;s not obvious if this batch should fail - I don&apos;t think it does currently. As I discussed with &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kaloian.manassiev&quot; class=&quot;user-hover&quot; rel=&quot;kaloian.manassiev&quot;&gt;kaloian.manassiev&lt;/a&gt;, we could probably tighten this up a bit by tracking within the ChunkManagerTargeter whether or not the collection&apos;s UUID has changed during the course of targeting. If it has, it seems pretty reasonable to abort the operation. At that point the shard key may have changed, and it&apos;s hard to say if the user wanted the operation to keep going.&lt;/p&gt;

&lt;p&gt;After speaking more to &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=asya&quot; class=&quot;user-hover&quot; rel=&quot;asya&quot;&gt;asya&lt;/a&gt;, it seems like the &quot;concurrent change in shard key&quot; problem isn&apos;t worth spending much effort solving. Instead, we should focus on the scenario where the mongos is stale. To achieve this, I propose we &lt;em&gt;always&lt;/em&gt; issue a listIndexes command to the primary shard for the $out&apos;s target database, and that this command should be versioned (attach a shard version to the command, which I checked does result in a StaleConfig exception if the mongos is stale). On a stale shard version error, we should retry the entire aggregate command, from the top. One of &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=nicholas.zolnierz&quot; class=&quot;user-hover&quot; rel=&quot;nicholas.zolnierz&quot;&gt;nicholas.zolnierz&lt;/a&gt;,s patches for&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-36047&quot; title=&quot;(Sharded target collection) Enforce existence of unique index containing fields of $out uniqueKey&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-36047&quot;&gt;&lt;del&gt;SERVER-36047&lt;/del&gt;&lt;/a&gt; already adds a listIndexes round-trip when a user specifies their own uniqueKey, so we aren&apos;t so worried about incurring that cost in general.&lt;/p&gt;

&lt;p&gt;How&apos;s that plan sound to everyone? cc &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kyle.suarez&quot; class=&quot;user-hover&quot; rel=&quot;kyle.suarez&quot;&gt;kyle.suarez&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=david.storch&quot; class=&quot;user-hover&quot; rel=&quot;david.storch&quot;&gt;david.storch&lt;/a&gt;&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="567554">SERVER-35954</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 4 Sep 2018 17:26:33 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 14 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/SERVER-35954'>SERVER-35954</a></s>, <s><a href='https://jira.mongodb.org/browse/WRITING-3246'>WRITING-3246</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-828</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 14 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>45.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>charlie.swanson@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>kaloian.manassiev@mongodb.com</customfieldvalue>
            <customfieldvalue>nicholas.zolnierz@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hu673r:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr8ibr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2466">Query 2018-09-10</customfieldvalue>
    <customfieldvalue id="2467">Query 2018-09-24</customfieldvalue>
    <customfieldvalue id="2487">Query 2018-10-08</customfieldvalue>
    <customfieldvalue id="2533">Query 2018-10-22</customfieldvalue>
    <customfieldvalue id="2534">Query 2018-11-05</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hu5td3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>