[SERVER-5923] Increase max document size to at least 64mb Created: 24/May/12 Updated: 14/Oct/23 Resolved: 05/Feb/19 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Performance, Storage |
| Affects Version/s: | 2.1.1 |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | John Wood | Assignee: | Asya Kamsky |
| Resolution: | Won't Fix | Votes: | 51 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
All platforms |
||
| Issue Links: |
|
||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||
| Description |
|
There are many systems that contain highly structured data (in my case loan data) containing historic data such as prices or versions. Being forced to split these into separate top-level documents because of the document size limitation in some ways makes the database no more useful than an RDBMS and defeats the purpose of using a document-oriented database. At the very least the document size should be increased from 16mb to a more realistic 64mb. In my systems, for example, a loan's data (with historic price information) is at around 30mb per loan, and could double. |
| Comments |
| Comment by Victor Parpoil [ 08/Feb/19 ] |
|
asya, Please reconsider this issue. Thanks |
| Comment by Asya Kamsky [ 05/Feb/19 ] |
|
Closing this ticket as we don't have plans to increase the size of document stored in the database. There is a separate ticket SERVER-12305 which is tracking allowing documents which represent commands, their responses, intermediate documents etc to be larger than 16MBs. |
| Comment by Victor Parpoil [ 05/Nov/18 ] |
|
+1 At least make this limit a conf parameter |
| Comment by Matt [ 26/Jul/17 ] |
|
+1 for this (or uncapping). Arbitrary limits are the devil. If I want to put a terabyte of ram in my system and query 500 mb documents, I should be able to, and not have the developers of mongodb trying to protect me from myself. |
| Comment by Salil Sabnis [ 08/Apr/17 ] |
|
We have the same issue of hitting 16MB limit. Is the max document size going to be increased ? |
| Comment by Bruno Bodson [ 08/Nov/16 ] |
|
+1. Using GridFS is not user-friendly because it stores the data in binary chunks. Redesigning my domain model will introduce lots of new references which will decrease performance. The same thing happens when increasing the document size limit. The user should be able make this trade-off by configuring the max document size limit. |
| Comment by Michael Dolinsky [ 05/Sep/16 ] |
|
It would be great if this could be configured and even if that puts us at risk of performance we would be accountable for that and test it... |
| Comment by Gintaras Bar [ 03/Feb/15 ] |
|
We have same issue hitting 16Mb per document. I could restructure in a different way (not what I want in a model), but even then it will hit016Mb limit in 3-4 years. |
| Comment by Konstantin Koshelyaev [ 28/Jan/15 ] |
|
I'm sure I need this. |
| Comment by guipulsar [ 06/Jan/15 ] |
|
this is an important things, we need more space too, 16mb is too limited |
| Comment by Rohit Badwaik [ 09/Sep/14 ] |
|
+1. |
| Comment by Ravi Kothiyal [ 15/Jul/14 ] |
|
We are also facing this issue . Love to have this feature. |
| Comment by Josh Hansen [ 14/Apr/14 ] |
|
A possible interim approach would be to throw a more specific exception than OperationFailure to better enable error handling |
| Comment by Josh Hansen [ 14/Apr/14 ] |
|
I just ran into this issue. I've been using mongo to aggregate events on a website by user, and over just a few days of activity some users had exceeded the 16mb threshold. The vast majority of users will never exceed the threshold, but for those that do it would be nice to have a smooth way of handling. Making this user configurable would be great, along with some guidance on how performance might be affected with higher or lower limits. |
| Comment by Itzhak Kagan [ 15/Feb/14 ] |
|
16 MB limitation is a real problem. within a year or two machines will have much more memory but the mongodb infrastructure will continue to regard it with old memory restrictions. |
| Comment by Joseph [ 22/Jul/13 ] |
|
How about changing in the source code and recompiling as a custom build that works with larger document size e.g. 64MB. Do committers think: A) it's just somewhat lowering the performance but it's going to work or B) There is a fundamental assumption (e.g. document size or memory issue) that makes more serious problems? |
| Comment by Ian Whalen (Inactive) [ 25/Jan/13 ] |
|
Chris, we don't have any plans to increase the limit (or make it configurable) right now, but as long as you're watching this ticket you'll get an email if/when we do schedule it. |
| Comment by James Blackburn [ 12/Nov/12 ] |
|
We're hitting 16mb limits here too. When storing time series data (even compressed, as blobs!) we inevitably need to break the data up by month to fit into the current 16MB limit. 16MB constrains the data layout significantly, as there's much additional complexity on the application side to stitch the underlying data back together. Currently we hit this limit a lot, and ideally it would be a tunable. (Note: we're sizing, and sharding our cluster such that data fits in memory on the servers.) |
| Comment by Eliot Horowitz (Inactive) [ 10/Sep/12 ] |
|
I don't think we'll just increase it again. |
| Comment by Christian Klukas [ 10/Sep/12 ] |
|
I have the same problem. In rare cases the size of documents may exceed the limits. I'd rather still save it somehow internally chunked or saved possibly more slowly but as a huge document, than to write work-around-code at every imaginable place. Also the proposed 64 MB could in some cases give a problem, so this limit needs to be user-configurable (in this case I would increase it much further, to really be save). I understand that there could be code errors, which could stay uncatched if there is no limit, but then I still would prefer it if this limit would be based on hardware limits (i.e. full disc) than on arbitrary limits. |