<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:01:14 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-2804] Transactional write-batch support</title>
                <link>https://jira.mongodb.org/browse/SERVER-2804</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;(Taken from &lt;a href=&quot;http://groups.google.com/group/mongodb-user/browse_thread/thread/cb38df80eac19a19&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://groups.google.com/group/mongodb-user/browse_thread/thread/cb38df80eac19a19&lt;/a&gt;)&lt;/p&gt;

&lt;p&gt;I&apos;d like to suggest adding &apos;write batch&apos; support to MongoDB in the future.&lt;/p&gt;

&lt;p&gt;Summary&lt;br/&gt;
----------------------------&lt;br/&gt;
It would be nice to be able to know a series of writes will all happen (eventual consistency, not necessarily atomic) - or not at all.&lt;/p&gt;

&lt;p&gt;Other transactional features like read locks etc are not to be supported with &apos;write batching&apos;.&lt;/p&gt;

&lt;p&gt;Detailed / Wall of text&lt;br/&gt;
----------------------------&lt;br/&gt;
I wanted to bring up a feature request regarding &apos;write transactions&apos; or &apos;atomic/eventually consistent write batch&apos; support.&lt;/p&gt;

&lt;p&gt;A big concern I have with a lack of &apos;transactions&apos; with MongoDB is that there can be a chance of data inconsistency.&lt;/p&gt;

&lt;p&gt;If the primary server dies mid way through updating/inserting multiple documents (perhaps across multiple shards), then you can get &apos;corrupt&apos; data (according to your application) by just missing a few vital documents in your domain model.&lt;/p&gt;

&lt;p&gt;Mongo&apos;s internal storage of the saved documents is fine; the only problem is that some document&apos;s didn&apos;t get saved before the crash, so the ones that were saved are not valid in a domain model sense because they are missing child documents etc.&lt;/p&gt;

&lt;p&gt;I understand transactions are not very desirable for performance reasons (especially due to locking).&lt;/p&gt;

&lt;p&gt;With that in mind, what about the concept of &apos;write batching&apos;, a write- only transaction, where all writes (across any number of shards) occur at once (eventually consistent) or not at all? Read locks are never taken; and you can&apos;t lock rows.&lt;/p&gt;

&lt;p&gt;For example; picking a simple example for demo purposes; I&apos;d want to write:&lt;/p&gt;

&lt;p&gt;&amp;#8211;&lt;br/&gt;
mongoDriver.StartBatch()&lt;br/&gt;
mongoDriver.db.users.insert(&lt;/p&gt;
{username: andrew}
&lt;p&gt;);&lt;br/&gt;
mongoDriver.db.messages.insert(&lt;/p&gt;
{message: &quot;Welcome!&quot;, username:andrew}
&lt;p&gt;);&lt;br/&gt;
mongoDriver.db.stats.update({$inc: &lt;/p&gt;
{ totalUsers: 1 }
&lt;p&gt;);&lt;br/&gt;
mongoDriver.CommitBatch(&lt;/p&gt;
{atomic: bool}
&lt;p&gt;)&lt;br/&gt;
&amp;#8211;&lt;/p&gt;

&lt;p&gt;I&apos;d like CommitBatch() to make sure that all of my writes in that batch are either committed; or none at all.&lt;/p&gt;

&lt;p&gt;The &apos;atomic&apos; flag in commit batch would decide whether a 2pc is used to ensure all commits occur at once across multiple shards. With atomic = False; commits happen without a 2PC in the sense that no co-ordination takes place; and an &apos;eventually consistent&apos; approach is taken to the commit (normally, these multi-shard changes would all appear in a few milliseconds anyway).&lt;/p&gt;

&lt;p&gt;This feature would let me know for certain that my data model will remain consistent (either I insert a new user, create a welcome message, and update my statistics &amp;#8211; or not at all).&lt;/p&gt;

&lt;p&gt;This feature CANNOT be used to perform read locking or &apos;bank balance&apos; transfers because you can&apos;t block readers trying to read a document mid &apos;write batch&apos; to evaluate the response - there is no read locking, your just applying a range of writes all at once or not at all.&lt;/p&gt;

&lt;p&gt;I don&apos;t see this impacting performance at all (especially with atomic: false); there are no locks taking place at any time, except for atomic: true which would introduce a slight delay when a 2pc is occurring to coordinate a write batch when requested in the rare case its needed.&lt;/p&gt;

&lt;p&gt;&amp;#8211;&lt;/p&gt;

&lt;p&gt;Conceptually I&apos;d imagine &apos;begin batch&apos; would mean each shard just logs any future write() queries to a local temporary collection (such as local.writebatches.&amp;lt;connection id&amp;gt;).&lt;/p&gt;

&lt;p&gt;A request to &apos;commit batch&apos; asks each shard whether they have finished writing to the local writebatch collection (or perhaps just always issue each writebatch insert with safe: true); if all is well, then a &apos;commit writebatch&apos; command is sent to each shard (without a 2pc, unless atomic: true was requested) to persist each write by looping over local.writebatches.&amp;lt;connection id&amp;gt; collection and really&lt;br/&gt;
performing the original request. &lt;/p&gt;

&lt;p&gt;Some thought needs to be put into failure handling (such as inserting a &apos;prepared&apos; flag to the local writebatch collection in the event of server failure to ensure its &quot;committed&quot; on recovery), but I think thats not too difficult. &lt;/p&gt;

&lt;p&gt;This would be a nice feature to have, it would prevent data inconsistency issues when you don&apos;t want your application to suffer; and avoids the locking associated with real transactions (that support read locks, isolation, etc).&lt;/p&gt;</description>
                <environment>n/a</environment>
        <key id="15191">SERVER-2804</key>
            <summary>Transactional write-batch support</summary>
                <type id="2" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14711&amp;avatarType=issuetype">New Feature</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="backlog-server-sharding">[DO NOT USE] Backlog - Sharding Team</assignee>
                                    <reporter username="plasma">Andrew Armstrong</reporter>
                        <labels>
                    </labels>
                <created>Mon, 21 Mar 2011 09:56:33 +0000</created>
                <updated>Tue, 6 Dec 2022 05:44:44 +0000</updated>
                            <resolved>Fri, 10 Aug 2018 16:56:35 +0000</resolved>
                                                                    <component>Usability</component>
                    <component>Write Ops</component>
                                        <votes>58</votes>
                                    <watches>52</watches>
                                                                                                                <comments>
                            <comment id="1972848" author="esha.maharishi@10gen.com" created="Fri, 10 Aug 2018 16:56:35 +0000"  >&lt;p&gt;Closing as a duplicate of the Cross-Shard Transactions project.&lt;/p&gt;</comment>
                            <comment id="1533130" author="mmaarouf" created="Mon, 27 Mar 2017 13:36:48 +0000"  >&lt;p&gt;This feature is &lt;b&gt;very&lt;/b&gt; important! Are there any plans to action this anytime soon?&lt;/p&gt;</comment>
                            <comment id="820668" author="fresheneesz" created="Tue, 3 Feb 2015 22:33:26 +0000"  >&lt;p&gt;I would love to see this kind of &quot;write batching&quot; support. I&apos;m also worried about catastrophic failures on the application side leading to inconsistent data inside the database.&lt;/p&gt;</comment>
                            <comment id="454491" author="chengas123" created="Mon, 11 Nov 2013 18:58:09 +0000"  >&lt;p&gt;Here are two related issues:&lt;br/&gt;
&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-11500&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-11500&lt;/a&gt;&lt;br/&gt;
&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-11508&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-11508&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Would be thrilled if this made its way into 2.8&lt;/p&gt;</comment>
                            <comment id="410278" author="elennaro" created="Fri, 23 Aug 2013 20:09:32 +0000"  >&lt;p&gt;Yes, its totally Important, and will do make MongoDB the leading NoSQL database. At least if you somehow add kind of journalled rollback operations for some write operations.&lt;br/&gt;
But the total Write Batch Transactional logic is totally required for business operations. For example my boss sad rough no to NoSQL due to this...&lt;/p&gt;</comment>
                            <comment id="171269" author="v.sevel@lombardodier.com" created="Thu, 4 Oct 2012 07:49:26 +0000"  >&lt;p&gt;I have an additional requirement around isolation: I need all updates to be visible at the same time. that is because I maintain financial positions. I cannot have in my system position1 that shows a $100 debit movement, then some time later position2 with a credit movement for the same amount. a financial transaction has to be complete or absent. eventually consistency is not enough for that type of use case.&lt;/p&gt;</comment>
                            <comment id="138392" author="ultralight32" created="Sun, 1 Jul 2012 11:58:38 +0000"  >&lt;p&gt;Well, I am trying to fix this problem using a table that keeps a big document for each batch before writing it to the database. Before reading any data this table must be empty, if not the batch must be performed (or one batch is in course). The problem resides about reading a half writting batch, so another locking system will be required.&lt;/p&gt;</comment>
                            <comment id="28578" author="ascobol" created="Mon, 11 Apr 2011 10:14:01 +0000"  >&lt;p&gt;This feature is very important to me too. I&apos;m designing a new project where a MongoDB database would fit very well, except for a few occasional cases where I need multi-document transactions. I don&apos;t know if this would impact performances at all, but even if this is the case it is not a problem for me as long as it&apos;s optional (and the penalty only occurs when used, ie it does not impact all requests but only the few ones that use this feature).&lt;/p&gt;</comment>
                            <comment id="28162" author="plasma" created="Thu, 7 Apr 2011 08:26:27 +0000"  >&lt;p&gt;I was just reading NoSQL @ Netflix (not using MongoDB; but the topic applies); one of the discussion points was how Netflix write a lot of &apos;consistency checker&apos; programs that run all the time (see &lt;a href=&quot;http://highscalability.com/blog/2011/4/6/netflix-run-consistency-checkers-all-the-time-to-fixup-trans.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://highscalability.com/blog/2011/4/6/netflix-run-consistency-checkers-all-the-time-to-fixup-trans.html&lt;/a&gt; ) to ensure the underlying data is (eventually) consistent due to the lack of write transactions.&lt;/p&gt;

&lt;p&gt;While developing a personal project using MongoDB; frequently I need to unfortunately reason about the chance that a crash/exception/etc on the client/server will &quot;corrupt&quot; the application data because one record was written but not the other.&lt;/p&gt;

&lt;p&gt;Without thinking too much into it; I would think Netflix&apos;s need to write &apos;consistency checkers&apos; and my own hesitation to do any multi-document updates would disappear if I could guarantee to my application that logical application data corruption is impossible (for all intents and purposes) by using a &apos;write batch&apos; feature that makes MongoDB assert to me that &quot;yes, these 2+ records your wanting to write/delete/upsert/etc will happen (eventually), or not at all&quot;.&lt;/p&gt;

&lt;p&gt;It&apos;s great MongoDB takes care of sharding, failover, replication etc easily. This is another big problem that could be tackled.&lt;/p&gt;

&lt;p&gt;I don&apos;t see any existing NoSQL solutions considering this as a real feature yet, is it important to anyone else?&lt;/p&gt;</comment>
                            <comment id="26450" author="plasma" created="Tue, 22 Mar 2011 00:23:50 +0000"  >&lt;p&gt;Hi Scott,&lt;/p&gt;

&lt;p&gt;Nope I didn&apos;t see that ticket.&lt;/p&gt;

&lt;p&gt;I&apos;m not sure whether that ticket expresses the other major part of this request, which is consistency of the write across multiple shards/documents, rather than just being say a perf optimization.&lt;/p&gt;

&lt;p&gt;Cheers&lt;/p&gt;</comment>
                            <comment id="26379" author="scotthernandez" created="Mon, 21 Mar 2011 12:50:07 +0000"  >&lt;p&gt;Did you see this: &lt;a href=&quot;http://jira.mongodb.org/browse/SERVER-2172&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;http://jira.mongodb.org/browse/SERVER-2172&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;I think in essences you are just asking for &quot;batching support&quot;, with a few special bits of behavior.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="96587">SERVER-11500</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="96622">SERVER-11508</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>11.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25141"><![CDATA[Sharding]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 21 Mar 2011 12:50:07 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 26 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>alexander.golin@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 26 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-server-sharding</customfieldvalue>
            <customfieldvalue>ascobol</customfieldvalue>
            <customfieldvalue>elennaro</customfieldvalue>
            <customfieldvalue>plasma</customfieldvalue>
            <customfieldvalue>chengas123</customfieldvalue>
            <customfieldvalue>fresheneesz</customfieldvalue>
            <customfieldvalue>esha.maharishi@mongodb.com</customfieldvalue>
            <customfieldvalue>ultralight32</customfieldvalue>
            <customfieldvalue>mmaarouf</customfieldvalue>
            <customfieldvalue>scotthernandez</customfieldvalue>
            <customfieldvalue>v.sevel@lombardodier.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrp33z:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr8otr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6585</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrkluf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>