<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:28:07 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-31766] Deprecate creation of new sharded output collection in mapReduce</title>
                <link>https://jira.mongodb.org/browse/SERVER-31766</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Starting in v3.6, creation of a new sharded output collections should be deprecated.&lt;/p&gt;

&lt;p&gt;When used, log a warning and bump a serverStatus counter so that we can track how often it&apos;s used.&lt;/p&gt;</description>
                <environment></environment>
        <key id="452731">SERVER-31766</key>
            <summary>Deprecate creation of new sharded output collection in mapReduce</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="backlog-server-query">Backlog - Query Team</assignee>
                                    <reporter username="dan@mongodb.com">Daniel Pasette</reporter>
                        <labels>
                            <label>qopt-team</label>
                    </labels>
                <created>Mon, 30 Oct 2017 17:12:32 +0000</created>
                <updated>Tue, 6 Dec 2022 03:47:58 +0000</updated>
                            <resolved>Wed, 27 Nov 2019 19:44:42 +0000</resolved>
                                                                    <component>MapReduce</component>
                                        <votes>0</votes>
                                    <watches>11</watches>
                                                                                                                <comments>
                            <comment id="2522931" author="nicholas.zolnierz" created="Thu, 7 Nov 2019 19:04:41 +0000"  >&lt;p&gt;Flagging for scheduling, I believe we have this covered by generically deprecating the &apos;out.sharded&apos; option.&lt;/p&gt;</comment>
                            <comment id="1799566" author="greg.mckeon" created="Thu, 8 Feb 2018 16:13:39 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=david.storch&quot; class=&quot;user-hover&quot; rel=&quot;david.storch&quot;&gt;david.storch&lt;/a&gt; based on an offline conversation with Kal, we&apos;ve found a workaround where we don&apos;t need to remove this in order to get rid of the distributed lock, so it is not a blocker for sharding anymore.&lt;/p&gt;

&lt;p&gt;Bumping the ticket back to query to triage whether or not to deprecate this capability in 4.0 for other reasons such as the difficulty with selecting sharding key.&lt;/p&gt;</comment>
                            <comment id="1789769" author="nicholas.zolnierz" created="Tue, 30 Jan 2018 21:14:29 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=greg.mckeon&quot; class=&quot;user-hover&quot; rel=&quot;greg.mckeon&quot;&gt;greg.mckeon&lt;/a&gt; this is certainly possible, the most logical place is in this function:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/master/src/mongo/s/commands/cluster_map_reduce_cmd.cpp#L635&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/master/src/mongo/s/commands/cluster_map_reduce_cmd.cpp#L635&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;IIRC, there are potential issues with this approach as there&apos;s a chance that mongos thinks the sharded output collection exists when it doesn&apos;t and vice-versa.  Are we OK with this sort of uncertainty when logging this? For the stats, it seemed much more relevant that the number be accurate. &lt;/p&gt;

&lt;p&gt;CC &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=david.storch&quot; class=&quot;user-hover&quot; rel=&quot;david.storch&quot;&gt;david.storch&lt;/a&gt; in case we want to bring this into the current sprint.&lt;/p&gt;</comment>
                            <comment id="1789731" author="greg.mckeon" created="Tue, 30 Jan 2018 20:51:32 +0000"  >&lt;p&gt;Tracking an offline conversation with Dan - it&apos;s ok if we just log a deprecation warning occasionally on any mapreduce a user runs (sharded or not), as long as the warning clarifies that it is specifically creation of new sharded output collection that is deprecated.  &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=nicholas.zolnierz&quot; class=&quot;user-hover&quot; rel=&quot;nicholas.zolnierz&quot;&gt;nicholas.zolnierz&lt;/a&gt; does that then make this possible?  If so, we&apos;d like to get a deprecation message in the docs and backported to 3.6 ASAP.&lt;/p&gt;

&lt;p&gt;CC: &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=schwerin&quot; class=&quot;user-hover&quot; rel=&quot;schwerin&quot;&gt;schwerin&lt;/a&gt; &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kaloian.manassiev&quot; class=&quot;user-hover&quot; rel=&quot;kaloian.manassiev&quot;&gt;kaloian.manassiev&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1724577" author="nicholas.zolnierz" created="Mon, 13 Nov 2017 21:44:03 +0000"  >&lt;p&gt;Moving back to the Query team backlog as this has fallen out of scope for 3.6.  Based on latest discussion (in-person and CR), tracking the counter only in mongos would be inaccurate as the sharding catalog cache is not guaranteed to be up-to-date.  One option to consider when revisiting this ticket is to modify the ConfigSvrShardCollectionCommand, which is sent from mongos to the configsvr, to include a detail in the response indicating whether the collection previously existed or not.  Additional information may be needed in the request to allow the configsvr to identify that the originating command is a cluster mapReduce through mongos.&lt;/p&gt;</comment>
                            <comment id="1721536" author="david.storch" created="Thu, 9 Nov 2017 15:34:17 +0000"  >&lt;p&gt;Per offline discussion, our plan is to deprecate the creation of a new sharded output collection, or a mapReduce to an existing &lt;em&gt;unsharded&lt;/em&gt; collection with the &lt;tt&gt;sharded:true&lt;/tt&gt; flag. In other words, we will deprecate mapReduce&apos;s ability to automatically shard and pre-split an output collection. Output to a preexisting sharded collection will not be deprecated, nor will output to a new unsharded collection. This description holds across all merge modes (replace, merge, and reduce).&lt;/p&gt;</comment>
                            <comment id="1716527" author="david.storch" created="Fri, 3 Nov 2017 15:19:32 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=nicholas.zolnierz&quot; class=&quot;user-hover&quot; rel=&quot;nicholas.zolnierz&quot;&gt;nicholas.zolnierz&lt;/a&gt;, your proposal for the new counter seems reasonable to me, though we should verify with downstream consumers of serverStatus() that their code will be tolerant of this addition. Looks like it should be fine to add a new &lt;tt&gt;ServerStatusMetricField&amp;lt;Counter64&amp;gt;&lt;/tt&gt; inside the map reduce command.&lt;/p&gt;

&lt;p&gt;The only remaining question I have is whether we need to deprecate the &lt;tt&gt;replace&lt;/tt&gt; mode with sharded output. As &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=schwerin&quot; class=&quot;user-hover&quot; rel=&quot;schwerin&quot;&gt;schwerin&lt;/a&gt; mentioned, this might not work correctly without being able to rename a sharded collection.&lt;/p&gt;</comment>
                            <comment id="1716415" author="nicholas.zolnierz" created="Fri, 3 Nov 2017 14:15:49 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=david.storch&quot; class=&quot;user-hover&quot; rel=&quot;david.storch&quot;&gt;david.storch&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=schwerin&quot; class=&quot;user-hover&quot; rel=&quot;schwerin&quot;&gt;schwerin&lt;/a&gt; As discussed in person, the first step is to understand the behavior of MR to an output collection with the various modes.  Below is a table of the tests I ran and the results (the top row represents the &quot;state&quot; of the output collection, and the 1st column is the mode). I used this &lt;a href=&quot;https://github.com/mongodb/mongo/blob/master/jstests/sharding/mapReduce_inSharded_outSharded.js&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;test&lt;/a&gt; as a reference:&lt;/p&gt;

&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;&amp;nbsp;&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;Non-existent&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;Sharded from output of previous MR&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;Sharded but with different shard key&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;Exists but not sharded&lt;/th&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;Merge&lt;/th&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;pass&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;pass&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;Fails to run if previous docs exist&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;Fails with incorrect count if previous docs exist&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;Replace&lt;/th&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;pass&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;pass&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;pass&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;pass&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;Reduce&lt;/th&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;pass&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;pass&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;Fails to run if previous docs exist&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;Fails with incorrect count if previous docs exist&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;


&lt;p&gt;As for the counter, I think the best place for it would be under &lt;tt&gt;metrics.commands.mapReduce.shardedOutNewCollection&lt;/tt&gt;, however I&apos;m not sure the counters/commands allow for arbitrary fields without including the counter in all commands.  Thoughts?&lt;/p&gt;</comment>
                            <comment id="1714086" author="david.storch" created="Tue, 31 Oct 2017 21:57:32 +0000"  >&lt;p&gt;Clarifying that we only intend to deprecate the creation of new sharded collections. Also, should this include deprecation of &lt;tt&gt;sharded: true&lt;/tt&gt; with the &lt;tt&gt;replace&lt;/tt&gt; action, regardless of whether the output collection already exists?&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="849999">SERVER-42134</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="350176">DOCS-9834</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25143"><![CDATA[Query]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 31 Oct 2017 20:15:22 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 13 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_14262" key="com.atlassian.jira.plugin.system.customfieldtypes:datepicker">
                        <customfieldname>End date</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 30 Oct 2017 23:59:59 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-766</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>alexander.golin@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 13 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-server-query</customfieldvalue>
            <customfieldvalue>dan@mongodb.com</customfieldvalue>
            <customfieldvalue>david.storch@mongodb.com</customfieldvalue>
            <customfieldvalue>greg.mckeon@mongodb.com</customfieldvalue>
            <customfieldvalue>nicholas.zolnierz@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htj0r3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr6r9b:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="1952">Query 2017-11-13</customfieldvalue>
    <customfieldvalue id="3287">Query 2019-12-16</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_14261" key="com.atlassian.jira.plugin.system.customfieldtypes:datepicker">
                        <customfieldname>Start date</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 30 Oct 2017 00:00:00 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htimuv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>