<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:52:29 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-39587] Include the final collection name in each oplog entry for commands using temporary collections</title>
                <link>https://jira.mongodb.org/browse/SERVER-39587</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;cross-DB renameCollection, mapreduce, and aggregate with $out all write to temporary collections that are eventually renamed after all the data has been inserted.&#160;&lt;/p&gt;

&lt;p&gt;For example, as of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-30371&quot; title=&quot;Separate renameCollection across DB commands into individual oplog entries&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-30371&quot;&gt;&lt;del&gt;SERVER-30371&lt;/del&gt;&lt;/a&gt;, a collection rename across databases generates these oplog entries:&#160;&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;a &lt;tt&gt;create&lt;/tt&gt;&#160;on the final database for a temporary collection name&lt;/li&gt;
	&lt;li&gt;&lt;tt&gt;insert&lt;/tt&gt; s to the temporary collection for all the documents&lt;/li&gt;
	&lt;li&gt;a &lt;tt&gt;renameCollection&lt;/tt&gt; from the temp collection to the final name&lt;/li&gt;
	&lt;li&gt;a &lt;tt&gt;drop&lt;/tt&gt;&#160;for the original collection name&#160;&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Looking at the &lt;tt&gt;create&lt;/tt&gt; and &lt;tt&gt;insert&lt;/tt&gt; operations, you cannot tell what the collection&apos;s final name will be.&lt;/p&gt;

&lt;p&gt;The oplog entries look similar for mapreduce and aggregation with $out.&lt;/p&gt;

&lt;p&gt;It would be helpful to mongomirror if the oplog entries included the final namespace.&lt;/p&gt;

&lt;p&gt;The reason is that&#160;MGOMIRROR-37 introduces the ability to only migrate/sync a subset of DBs and/or collections.&lt;/p&gt;

&lt;p&gt;Looking at the &lt;tt&gt;create&lt;/tt&gt; and &lt;tt&gt;insert&lt;/tt&gt; oplog entries on the temporary collection, we cannot tell if the final namespace matches the user provided filter, and if we should be applying the ops or not.&lt;br/&gt;
 &#160;&lt;br/&gt;
 For now, we can handle this by always applying entries for temporary collections on a database that is fully/partially included in the filter, and then dropping the temporary namespace if we get to the &lt;tt&gt;renameCollection&lt;/tt&gt; op and it turns out the final name is not one we are mirroring. But that ends up being a lot of extra work for mongomirror. Additionally, it requires us to hard-code in what format temporary collection names are in, which seems risky if there is any chance those could change in future server versions. If the &lt;tt&gt;create&lt;/tt&gt; and &lt;tt&gt;insert&lt;/tt&gt;&#160;had a field indicating the final namespace, that would&lt;/p&gt;

&lt;p&gt;a) tell us that the op is definitely on a temp collection (this would continue to work even if the temp name format changed), and&lt;/p&gt;

&lt;p&gt;b) allow us to determine whether we care about the final namespace and should apply the op or skip it&lt;/p&gt;</description>
                <environment></environment>
        <key id="697756">SERVER-39587</key>
            <summary>Include the final collection name in each oplog entry for commands using temporary collections</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="12300">Won&apos;t Do</resolution>
                                        <assignee username="backlog-server-execution">Backlog - Storage Execution Team</assignee>
                                    <reporter username="kaitlin.mahar@mongodb.com">Kaitlin Mahar</reporter>
                        <labels>
                    </labels>
                <created>Thu, 14 Feb 2019 18:57:13 +0000</created>
                <updated>Tue, 6 Dec 2022 03:05:43 +0000</updated>
                            <resolved>Tue, 12 May 2020 21:47:55 +0000</resolved>
                                                                    <component>Storage</component>
                                        <votes>0</votes>
                                    <watches>11</watches>
                                                                                                                <comments>
                            <comment id="3081946" author="milkie" created="Tue, 12 May 2020 20:55:16 +0000"  >&lt;p&gt;As Geert mentioned, we are replicating based on collection UUIDs now and the ns field is just informational.  It sounds like we should switch MongoMirror to do replication filtering based on UUID, since those do not change on rename.  This would solve the $out temp collection rename issue described above.&lt;/p&gt;</comment>
                            <comment id="3081245" author="geert.bosch" created="Tue, 12 May 2020 16:53:00 +0000"  >&lt;p&gt;Currently, the only notable property of temporary collections is that they are deleted on restart or replica-set failover. As it is our clear objective to make such events unnoticeable, so the entire concept of temporary collections will go away at some point.&lt;/p&gt;

&lt;p&gt;Their main current use is as an approximation of atomically replacing a collection&#160;(using the &lt;tt&gt;aggregate&lt;/tt&gt; command with &lt;tt&gt;$out&lt;/tt&gt;, for example), but the isolation is fragile as it depends on cooperation from other readers and writers treating these collection names as special and hidden. Furthermore, there is nothing special about the rename done at the end of such an &lt;tt&gt;aggregate&lt;/tt&gt; command. A user might well output to a collection &lt;tt&gt;sales.report&lt;/tt&gt; and then rename that collection to &lt;tt&gt;sales.report2020Q2&lt;/tt&gt;, so the proposed addition would not work in general. Or to reverse the logic: any solution that can deal with a user renaming an existing collection should also be able to deal with renaming temporary collections.&lt;/p&gt;

&lt;p&gt;Since MongoDB 3.6, replication has been based on collection UUIDs rather than collection names, as it proved impossible to correctly sync nodes in the presence of concurrent renames due to idempotency issues. So, while each oplog entry currently still has a &lt;tt&gt;ns&lt;/tt&gt; field with the collection name these are merely informational at this point and for backward compatibility with existing tools and possibly reading the oplog. Adding more special handling for temporary collections and collection names seems a step in the wrong direction.&lt;/p&gt;</comment>
                            <comment id="3057254" author="david.golden" created="Mon, 27 Apr 2020 18:55:58 +0000"  >&lt;p&gt;FYI, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kaitlin.mahar&quot; class=&quot;user-hover&quot; rel=&quot;kaitlin.mahar&quot;&gt;kaitlin.mahar&lt;/a&gt; is not working on mongomirror anymore.&#160; The current lead is &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ryan.chipman%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;ryan.chipman@mongodb.com&quot;&gt;ryan.chipman@mongodb.com&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;I&apos;d like to have this reconsidered.&#160; In the original ticket description it says:&lt;/p&gt;

&lt;blockquote&gt;&lt;p&gt;it requires us to hard-code in what format temporary collection names are in, which seems risky if there is any chance those could change in future server versions&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;We have tightly coupled behavior between the server and mongomirror based on a collection name format.  If the decision is not to change the oplog per this ticket, does the server have tests to verify that the temporary collection name format remains unchanged?  If not, it would be a good idea to add such tests.&lt;/p&gt;</comment>
                            <comment id="2188886" author="asya" created="Fri, 22 Mar 2019 17:30:42 +0000"  >&lt;p&gt;Is this specific to cross database rename/output?  Aggregation $out does not support specifying a different database at least up through 4.0 (though there&apos;s a plan to support that in the future).&lt;/p&gt;</comment>
                            <comment id="2158772" author="milkie" created="Thu, 21 Feb 2019 18:59:04 +0000"  >&lt;p&gt;Also, what will you do if a multi-document transaction contains operations that are partially filtered?&lt;/p&gt;</comment>
                            <comment id="2158762" author="milkie" created="Thu, 21 Feb 2019 18:53:35 +0000"  >&lt;p&gt;How are you planning on handling a renameCollection within the same database, if one collection is in the filter and one is out?&lt;/p&gt;</comment>
                            <comment id="2153675" author="kaitlin.mahar" created="Fri, 15 Feb 2019 18:59:13 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=schwerin&quot; class=&quot;user-hover&quot; rel=&quot;schwerin&quot;&gt;schwerin&lt;/a&gt;:&#160;Ah, I didn&apos;t realize those both use temporary collections - thanks for pointing that out! The same issue applies in those cases as well. Updating title/description accordingly.&lt;/p&gt;</comment>
                            <comment id="2153090" author="schwerin" created="Fri, 15 Feb 2019 13:34:08 +0000"  >&lt;p&gt;Is this different from an aggregation or mapreduce that outputs to a temporary collection and then renames it? Is rename special?&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="409115">SERVER-30371</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1348930">SERVER-48165</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25136"><![CDATA[Storage Execution]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 15 Feb 2019 13:34:08 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 39 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>false</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>alexander.golin@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 39 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>schwerin@mongodb.com</customfieldvalue>
            <customfieldvalue>asya.kamsky@mongodb.com</customfieldvalue>
            <customfieldvalue>backlog-server-execution</customfieldvalue>
            <customfieldvalue>david.golden@mongodb.com</customfieldvalue>
            <customfieldvalue>milkie@mongodb.com</customfieldvalue>
            <customfieldvalue>geert.bosch@mongodb.com</customfieldvalue>
            <customfieldvalue>kaitlin.mahar@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hunl5z:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr5h4n:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hun7fb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>