[SERVER-36330] Total transaction operations size only 16MB? Created: 27/Jul/18  Updated: 27/Oct/23  Resolved: 30/Jul/18

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: 4.0.0
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Vyacheslav Stroy Assignee: Alyson Cabral (Inactive)
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

NodeJS driver v3.1.0
Windows 10 x64


Issue Links:
Backports
backports SERVER-36428 WriteConflict inside a transaction on... Closed
Participants:

 Description   

In my application I'm trying to make use of newly added multi-document transactions feature.

I got the following error:

Total size of all transaction operations must be less than 16793600. Actual size is 16793817

Why this limit was not specified in documentation? And do you have plans to increase the limit?

During testing I also run into several other problems, like unexpected "Transaction aborted" errors when I had a few thousands of simultaneously open server sessions. Is there any document (like blogpost or maybe some internal docs) with a description of all limitations?



 Comments   
Comment by Grigori Melnik (Inactive) [ 24/Jul/19 ]

Check out v4.2 which supports large transactions over multiple oplog entries.

Comment by Rupesh Mishra [ 07/Jun/19 ]

 Hi Aly, 

Is it not that we can change the oplog size as mentioned in the below link 

https://docs.mongodb.com/manual/tutorial/change-oplog-size/

If yes, then does it means we can overcome this transaction limit size by changing the oplog size?

If no, then are oplogs mentioned in the above link is different than the oplog entry which we are discussing in this thread so far? If so, what is the exact difference between both these oplogs?

Comment by Kelvin Muia [ 16/May/19 ]

Any update on removing the transaction size constraint

Comment by Alyson Cabral (Inactive) [ 22/Apr/19 ]

Hi rupesm1993,

In 4.0, each transaction has an oplog entry that consists of all the deltas changed to the document (or row), not the full document (or row). For example, if you're doing many small updates within a single transaction, only the fields changed will be represented in the 16MB limit oplog entry. However if you're doing many inserts within a single transaction, each of the full documents will be stored in that 16MB entry.

We are actively working on removing this size constraint.

Comment by Rupesh Mishra [ 17/Apr/19 ]

Hi, We have 1 million  rows in MySQL and we want to migrate to MongoDB, but this limit of 16 MB inside the transaction is making MongoDB impractical I guess. Because if any one want to use transaction, probability of data having greater than 16 MB is always greater.

 

Let me know if I am missing something as per the docs it's the document size which is 16 MB and document means rows in MySQL and my rows size is no where greater than 16 MB. 

If this size is sum of the all rows in transaction, can you know if anyone has been able to use this transaction feature with any data greater than some million rows?

Even if it will change to 64 MB, will it be of any help. Kindly help

Comment by Alyson Cabral (Inactive) [ 27/Sep/18 ]

Not just yet. Please join the beta group linked in my previous comment and I'll notify you as soon as it's available to test.

Comment by daniel moqvist [ 27/Sep/18 ]

Hi, today we encountered the 16Mb limitaton in our transactions. 

Do you have any updated status about the fix for this transaction limitation?

Is there any beta or rc we can test?

Comment by Alyson Cabral (Inactive) [ 30/Jul/18 ]

I'm going to close this ticket as works as designed but feel free to open a separate server ticket if you do find unexpected behavior.

Also, I recommend you join the transactions beta group and ask any development questions related to transactions.

Comment by Vyacheslav Stroy [ 27/Jul/18 ]

Hey @Aly,

Thanks for the reference and clarification of implementation details.

Will get back to you with detailed stack traces once I run into the mentioned error again.

Comment by Alyson Cabral (Inactive) [ 27/Jul/18 ]

Hi there,

This limit is outlined in the documentation under oplog size limit on the production considerations page.

In MongoDB 4.0 the entire description of modifications in a transaction (i.e. changes made by each statement) are represented as a single oplog entry. Not even oplog entries are immune to 16MB document size limit upheld by the system. While this limitation does encourage developers to conform to MongoDB best practices in some cases, it is a hard limitation to reason about, as the combined size of the description of changes should be an implementation detail for many users. It is a priority for us to remove this limitation by allowing transactions to be represented across multiple oplog entries.

I'll update the transactions beta group as soon as we have the functionality available to test. Also, feel free to ask any transactions usability questions there. I'd love to hear more about the unexpected aborted transactions you are encountering. Keep in mind that transactions are aborted when they encounter write conflicts, so if you are doing concurrent writes to the same set of documents that could be a reason.

As an additional resource, I gave a talk at MongoDB World that goes over limitations and guarantees.

Aly Cabral
MongoDB Product Manager, Distributed Systems

Generated at Thu Feb 08 04:42:47 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.