<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:08:02 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-5149] Auto-generated short-lived collections for output from $merge or $out</title>
                <link>https://jira.mongodb.org/browse/SERVER-5149</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;It would be useful to have the ability to store results from your computations in the server for future inspection in a way that would go away after some period of time (perhaps when the session ends?) without having to choose unique names.&lt;/p&gt;

&lt;h3&gt;&lt;a name=&quot;OriginalDescription&quot;&gt;&lt;/a&gt;Original Description&lt;/h3&gt;
&lt;p&gt;With mongo 1.8 the ability to have a mapreduce write its results to a temporary table was removed, according to the documentation here: &lt;a href=&quot;http://www.mongodb.org/display/DOCS/MapReduce#MapReduce-Outputoptions&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://www.mongodb.org/display/DOCS/MapReduce#MapReduce-Outputoptions&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;This means that clients that want &apos;one shot&apos; MR operations are forced to either use inline output, or to manage the naming, creation, and reaping of the output collection. If, as is &lt;b&gt;very often&lt;/b&gt; the case, the total data size that will be created exceeds the current database maximum BSON object size (16MB these days), then you cannot use inline, and the only option is for the client to manually manage the lifetime of the output collection. However, this is not as easy as it sounds:&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;The client must have some mechanism for generating unique names for the output collection. While not necessarily hard, this is annoying. It also creates a risk that less than diligent authors will pick a &apos;unique&apos; name like &apos;my_secret_output_collection&apos;, which then of course will inevitably collide months later when some other genius picks the same &apos;unique&apos; name. However, the database server can easily construct non-colliding temporary collection names and communicate the name back to the client in the MR response. Even if you don&apos;t restore the server side reaping feature, the server could still synthesize a safe unique name for you.&lt;/li&gt;
&lt;/ul&gt;


&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;The client needs to carry about the name of the output collection somewhere so that it can (try its best to) drop the collection later. This frustrates certain programming idioms like iterating the cursor from a coroutine/generator, since there now needs to be a post-iteration action that drops the collection.&lt;/li&gt;
&lt;/ul&gt;


&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;The client needs to drop the collection when it is done with it, but is unequipped to reliably do so. No matter how many exception handlers, signal handlers, finalizers, or redundant power supplies and switches you pollute your code and infrastructure with, there is always a way that the client fails to honor its obligation to drop the collection. On the other hand, the mongodb server is in a uniquely qualified position to reap stale temporary collections, since it can easily detect that the client responsible for the collection has gone away and that it should now reap the useless collection. In addition, the server can much more easily work around the sort of failure modes noted above, since in the event of its own crash, it should easily be able to identify temporary collections on restart/shutdown and destroy them.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Overall, I&apos;m perplexed why this very useful feature was removed. It takes away something that the server should be easily able to do with very high reliability, and forces clients to make complex (but ultimately hopeless) efforts to re-implement the feature themselves.&lt;/p&gt;</description>
                <environment></environment>
        <key id="31878">SERVER-5149</key>
            <summary>Auto-generated short-lived collections for output from $merge or $out</summary>
                <type id="2" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14711&amp;avatarType=issuetype">New Feature</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="backlog-query-execution">Backlog - Query Execution</assignee>
                                    <reporter username="andrew.morrow@mongodb.com">Andrew Morrow</reporter>
                        <labels>
                            <label>query-44-grooming</label>
                    </labels>
                <created>Wed, 29 Feb 2012 22:48:52 +0000</created>
                <updated>Tue, 6 Dec 2022 05:35:36 +0000</updated>
                            <resolved>Fri, 4 Feb 2022 15:09:18 +0000</resolved>
                                    <version>1.8.0</version>
                    <version>2.0.0</version>
                                                    <component>Aggregation Framework</component>
                    <component>MapReduce</component>
                                        <votes>1</votes>
                                    <watches>8</watches>
                                                                                                                <comments>
                            <comment id="4335992" author="esha.bhargava" created="Fri, 4 Feb 2022 15:08:27 +0000"  >&lt;p&gt;Closing these tickets as part of the deprecation of mapReduce.&lt;/p&gt;</comment>
                            <comment id="2740768" author="pawel.terlecki" created="Sat, 18 Jan 2020 21:33:31 +0000"  >&lt;p&gt;Support for temp collections is fundamental if one wants to Mongo to act as a node in any cross-database federated processing. For example, ETL in Alteryx, Tableau and basically any other cross-db processing engine is the only way to move piece of data across databases, e.g. for joins. Without this feature data from mongo will always need to be extracted fully for processing, even if most data in the scenario is in Mongo. This will happen in live federated models involving mongo and other databases in Tableau. Some of data blending scenarios will be completely unavailable.&lt;/p&gt;

&lt;p&gt;In addition, large filters are often externalized by temp collections for fast filtering. E.g. Tableau is extremely slow against data sources that do not support temp tables. In our case, we first need to fix our lookups with SBE to actually be faster in this scenario.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2383335" author="charlie.swanson" created="Thu, 22 Aug 2019 20:35:33 +0000"  >&lt;p&gt;Converting this to a feature request since it&apos;s been long enough since that release that I wouldn&apos;t consider this a bug anymore.&lt;/p&gt;</comment>
                            <comment id="2188897" author="asya" created="Fri, 22 Mar 2019 17:35:38 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=acm&quot; class=&quot;user-hover&quot; rel=&quot;acm&quot;&gt;acm&lt;/a&gt; is this an issue? the way mr (and agg) output now it&apos;s written to a temp collection that&apos;s then renamed, so is that sufficient?  I&apos;m not sure what pre 1.8 behavior was.&lt;/p&gt;</comment>
                            <comment id="94043" author="eliot" created="Thu, 1 Mar 2012 05:07:08 +0000"  >&lt;p&gt;The major problem was that the previous incarnation didn&apos;t make a lot of sense as designed.&lt;/p&gt;

&lt;p&gt;Agree it is a good feature, just not sure on all the design issues at this point.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25125"><![CDATA[Query Execution]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 1 Mar 2012 05:07:08 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>alexander.golin@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>andrew.morrow@mongodb.com</customfieldvalue>
            <customfieldvalue>asya.kamsky@mongodb.com</customfieldvalue>
            <customfieldvalue>backlog-query-execution</customfieldvalue>
            <customfieldvalue>charlie.swanson@mongodb.com</customfieldvalue>
            <customfieldvalue>eliot</customfieldvalue>
            <customfieldvalue>esha.bhargava@mongodb.com</customfieldvalue>
            <customfieldvalue>pawel.terlecki@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrobi7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr2h0v:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6057</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht0txb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>