[SERVER-5923] Increase max document size to at least 64mb Created: 24/May/12  Updated: 14/Oct/23  Resolved: 05/Feb/19

Status: Closed
Project: Core Server
Component/s: Performance, Storage
Affects Version/s: 2.1.1
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: John Wood Assignee: Asya Kamsky
Resolution: Won't Fix Votes: 51
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

All platforms


Issue Links:
Duplicate
is duplicated by SERVER-8047 Increase the limit for JsonArray that... Closed
is duplicated by SERVER-46212 Make BSON Object Max Size customizable Closed
Related
related to SERVER-60040 Increase max document size to at leas... Closed
related to SERVER-12305 Allow command request and response BS... Backlog
related to SERVER-27013 Allow results of the aggregation fram... Closed
is related to SERVER-23482 Should the 16Mb limit be configurable? Closed
Participants:

 Description   

There are many systems that contain highly structured data (in my case loan data) containing historic data such as prices or versions. Being forced to split these into separate top-level documents because of the document size limitation in some ways makes the database no more useful than an RDBMS and defeats the purpose of using a document-oriented database.

At the very least the document size should be increased from 16mb to a more realistic 64mb. In my systems, for example, a loan's data (with historic price information) is at around 30mb per loan, and could double.



 Comments   
Comment by Victor Parpoil [ 08/Feb/19 ]

asya, Please reconsider this issue.
Mongo DB was at some point limited to 4MB and yet you finally decided to increase to 16MB at some point (read again the discussion on issue 431 (https://jira.mongodb.org/browse/SERVER-431) that was back in 2009 !). We are now 10 years later. Data is heavier. System performance has increased a lot. All that has been written is still valid as of today.

Thanks

Comment by Asya Kamsky [ 05/Feb/19 ]

Closing this ticket as we don't have plans to increase the size of document stored in the database.

There is a separate ticket SERVER-12305 which is tracking allowing documents which represent commands, their responses, intermediate documents etc to be larger than 16MBs.

Comment by Victor Parpoil [ 05/Nov/18 ]

+1 At least make this limit a conf parameter

Comment by Matt [ 26/Jul/17 ]

+1 for this (or uncapping). Arbitrary limits are the devil. If I want to put a terabyte of ram in my system and query 500 mb documents, I should be able to, and not have the developers of mongodb trying to protect me from myself.

Comment by Salil Sabnis [ 08/Apr/17 ]

We have the same issue of hitting 16MB limit. Is the max document size going to be increased ?

Comment by Bruno Bodson [ 08/Nov/16 ]

+1. Using GridFS is not user-friendly because it stores the data in binary chunks. Redesigning my domain model will introduce lots of new references which will decrease performance. The same thing happens when increasing the document size limit. The user should be able make this trade-off by configuring the max document size limit.

Comment by Michael Dolinsky [ 05/Sep/16 ]

It would be great if this could be configured and even if that puts us at risk of performance we would be accountable for that and test it...

Comment by Gintaras Bar [ 03/Feb/15 ]

We have same issue hitting 16Mb per document. I could restructure in a different way (not what I want in a model), but even then it will hit016Mb limit in 3-4 years.

Comment by Konstantin Koshelyaev [ 28/Jan/15 ]

I'm sure I need this.

Comment by guipulsar [ 06/Jan/15 ]

this is an important things, we need more space too, 16mb is too limited

Comment by Rohit Badwaik [ 09/Sep/14 ]

+1.
We are reaching the limit when trying to get data for 2 delears. Afraid of what will happen with growing/changing client requirements.

Comment by Ravi Kothiyal [ 15/Jul/14 ]

We are also facing this issue . Love to have this feature.

Comment by Josh Hansen [ 14/Apr/14 ]

A possible interim approach would be to throw a more specific exception than OperationFailure to better enable error handling

Comment by Josh Hansen [ 14/Apr/14 ]

I just ran into this issue. I've been using mongo to aggregate events on a website by user, and over just a few days of activity some users had exceeded the 16mb threshold. The vast majority of users will never exceed the threshold, but for those that do it would be nice to have a smooth way of handling. Making this user configurable would be great, along with some guidance on how performance might be affected with higher or lower limits.

Comment by Itzhak Kagan [ 15/Feb/14 ]

16 MB limitation is a real problem. within a year or two machines will have much more memory but the mongodb infrastructure will continue to regard it with old memory restrictions.
I think that 64 MB document limitation will not be enough for some applications as well and I wouldn't assume from that that it's a bad data model. It takes Just one document that oversize that limitation that can ruin your model. I think that it's better to know that for some situations there will performance issues , assuming that they'll be the exceptions rather than the general cases.
I think that it should be configurable or have a much broader limit.

Comment by Joseph [ 22/Jul/13 ]

How about changing in the source code and recompiling as a custom build that works with larger document size e.g. 64MB. Do committers think: A) it's just somewhat lowering the performance but it's going to work or B) There is a fundamental assumption (e.g. document size or memory issue) that makes more serious problems?

Comment by Ian Whalen (Inactive) [ 25/Jan/13 ]

Chris, we don't have any plans to increase the limit (or make it configurable) right now, but as long as you're watching this ticket you'll get an email if/when we do schedule it.

Comment by James Blackburn [ 12/Nov/12 ]

We're hitting 16mb limits here too.

When storing time series data (even compressed, as blobs!) we inevitably need to break the data up by month to fit into the current 16MB limit. 16MB constrains the data layout significantly, as there's much additional complexity on the application side to stitch the underlying data back together.

Currently we hit this limit a lot, and ideally it would be a tunable.

(Note: we're sizing, and sharding our cluster such that data fits in memory on the servers.)

Comment by Eliot Horowitz (Inactive) [ 10/Sep/12 ]

I don't think we'll just increase it again.
If we change it, it would be to make it configurable or something else.

Comment by Christian Klukas [ 10/Sep/12 ]

I have the same problem. In rare cases the size of documents may exceed the limits. I'd rather still save it somehow internally chunked or saved possibly more slowly but as a huge document, than to write work-around-code at every imaginable place. Also the proposed 64 MB could in some cases give a problem, so this limit needs to be user-configurable (in this case I would increase it much further, to really be save). I understand that there could be code errors, which could stay uncatched if there is no limit, but then I still would prefer it if this limit would be based on hardware limits (i.e. full disc) than on arbitrary limits.

Generated at Thu Feb 08 03:10:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.