[SERVER-2172] Support Batching Mixed Operations Against Same Collection Created: 02/Dec/10  Updated: 06/Dec/22

Status: Backlog
Project: Core Server
Component/s: Usability
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Minor - P4
Reporter: Scott Hernandez (Inactive) Assignee: Backlog - Query Optimization
Resolution: Unresolved Votes: 60
Labels: oneshottxn
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-4004 Bulk Upsert Closed
is depended on by SERVER-20211 Cross collection batch operation Backlog
Related
is related to SERVER-4004 Bulk Upsert Closed
Assigned Teams:
Query Optimization
Participants:

 Description   

Allow mixed operations (insert, update, delete) on the same collection to be sent to the server in a single write command.



 Comments   
Comment by Asya Kamsky [ 15/May/17 ]

There are multiple things that are discussed in the comments of this ticket, some of which are already implemented, and others may be tracked by different tickets. I'm attempting to summarize the state of it here:

  • Sending batches of same operations against a single collection: implemented in 2.6 SERVER-4004
  • Sending batches of mixed operations against same collection: tracked in this ticket
  • Batched operations against separate collections: SERVER-20211
    this was initially marked as duplicate of this ticket. I re-opened it to track that feature separately, and made it dependent on this ticket being implemented/resolved
  • All-or-nothing batches of operations, also known as transactions are out of scope for this ticket and are tracked separately by SERVER-2804.

I updated description of this ticket to clarify that it's only tracking sending batches of mixed operations for a single collection (inserts, updates and removes).

Comment by Nick Judson [ 08/Sep/15 ]

Adding my name to the list who would find this useful.

Comment by Derek Milne [ 05/Sep/15 ]

For me this is the most needed feature in mongodb. To be unable to do transactions forces you to write complex transactions yourself or switch to SQL.

Comment by Kevin J. Rice [ 25/Feb/13 ]

I would like to vote for a simple version of this: BATCH UPDATES. That is, I'm issuing thousands of update statements a second. I would like to send them as a list of updates rather than do them one by one, with the latency involved in that.

Currently there is support for batch inserts. This feature may tie-in with that functionality.

Playing devil's advocate to my own argument, I can see a possible complication. I have a sharded, replicated collection. So, let's say I send in a list of updates (each of a single separate document indexed by _id). The router (mongos) would have to split the updates to separate lists to go to their respective shards.

Other than that, it seems a straightforward performance increaser.

Re: failures, handle it the same as batch inserts, insert all possible records/documents and return data on which ones failed, or whatever is the easiest functionality to implement and I'll cope with the downsides.

Comment by John Wood [ 13/Sep/12 ]

Allowing batch queries would also resolve SERVER-432 (server side expansion of DbRefs), and make it much easier to create a query preprocessor that would efficiently allow for more complex queries and eliminate some limitations (eg. SERVER-5937 (splitting documents to overcome size limitations)).

I can imagine it would be hard because you would need to support multiple cursors per connection, but even if initially you just limited it to returning all the data in one BSON it would still reap a lot of benefits.

Comment by Moshe [ 12/Aug/12 ]

@Caleb, I agree. Fail-safe systems are a nightmare to build with MongoDB. Here described a problem that is similar to mine: can't roll back a set of writes (insertions/updates) when one failed. I needed this in several applications, but I gave up and left them non-fail-safe. I imagine that's what most people do when they encounter this problem, but they don't when the worst possible failure is a disaster (like a payment that was not properly rewarded). Then they do a workaround.

The workarounds are terrible. The most complex part of the application becomes the database, which was the simplest before.

Why won't you admit that's a serious problem?

Comment by Caleb Jones [ 24/Jul/12 ]

I disagree with the "Minor" Priority. This would be a huge feature add similar in impact to the aggregation framework.

Probably the #1 complaint or deterrent people cite with MongoDB is the lack of JOINs. Yes, that is part of the trade-offs with the solution. But when de-normalization is undesirable, being able to perform a batch of operations would avoid the bursting of one-by-one commands in loops and could overcome some of the reservations people have in adopting MongoDB.

Another use case impacted by this:
I'm currently working on processing a stream of data to convert into mongo insert/update/delete commands. I was disappointed that I have to fire off the commands one at at time instead of queuing them up into batches to reduce round-trips to the server.

Hopefully this can get a bump in priority.

Generated at Thu Feb 08 02:59:10 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.