[SERVER-1153] Create an in-memory only storage engine Created: 25/May/10 Updated: 22/Mar/17 Resolved: 09/Nov/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Storage |
| Affects Version/s: | None |
| Fix Version/s: | 3.2.0-rc2 |
| Type: | New Feature | Priority: | Major - P3 |
| Reporter: | phpMoAdmin | Assignee: | Geert Bosch |
| Resolution: | Done | Votes: | 101 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||
| Sprint: | QuInt C (11/23/15) | ||||||||||||||||||||||||
| Participants: |
Alex Barnes, Antoine Girbal, auto, Ben McCann, Chad Kreimendahl, Dominique De Vito, Eliot Horowitz, Evgeny Myasishchev, Geert Bosch, Githook User, guipulsar, Jean-Pierre Bergamin, Markus Haslinger, Oded Maimon, Phil Idem, phpMoAdmin, Rakesh Madhavakurup, Remon van Vliet, Richard Kreuter, Ron Avnur, Scott Lyons, Tim Niemueller, Victor Andritoiu
|
||||||||||||||||||||||||
| Description |
|
Mongo is already a performance leader, but it would be great to push this farther via an option for storage entirely within ramspace like an Oracle In-Memory Database or MySQL memory table (aka. heap table) This would also need to turn off the disk-space pre-allocation; with storage in RAM you would not need any pre-allocation since re-allocating additional space would be an almost instant task. I have created a hack for this that works but even with nssize=2 and smallfiles set the default disk pre-allocation makes my setup rather limited in use. |
| Comments |
| Comment by Geert Bosch [ 09/Nov/15 ] |
|
The enterprise build contains a new in-memory storage engine for production use. It is currently in beta and is available in the enterprise build since 3.2.0-rc2. To enable, use --storageEngine inMemory. More documentation coming soon in the release notes for v3.2. The enterprise server is openly available for free trial and development. You may use it in development / eval without a commercial license (https://www.mongodb.com/lp/download/mongodb-enterprise). For the watchers of this ticket who are looking for a non-persistant storage engine for testing purposes, we have provided a storage engine for this use case in the community build. Use --storageEngine ephemeralForTest to enable an in-memory solution which supports collection level locking. As the name of this option suggests, do not use it in production. |
| Comment by Githook User [ 02/Nov/15 ] |
|
Author: {u'username': u'GeertBosch', u'name': u'Geert Bosch', u'email': u'geert@mongodb.com'}Message: |
| Comment by Githook User [ 02/Nov/15 ] |
|
Author: {u'username': u'GeertBosch', u'name': u'Geert Bosch', u'email': u'geert@mongodb.com'}Message: |
| Comment by Githook User [ 02/Nov/15 ] |
|
Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}Message: |
| Comment by Githook User [ 16/Oct/15 ] |
|
Author: {u'username': u'monkey101', u'name': u'Dan Pasette', u'email': u'dan@10gen.com'}Message: Revert " This reverts commit e0001f87181b6b195760ccd872e719a399475778. |
| Comment by Chad Kreimendahl [ 16/Oct/15 ] |
|
The fact that it's enterprise only, given how high it's been rated among everyone is more curious than the notifications, Tim |
| Comment by Tim Niemueller [ 15/Oct/15 ] |
|
It's rather frustrating to even be bothered by notifications about commits to a non-publicly accessible repository that implements a feature that obviously many (on the open source or academic side) need – to say the least. |
| Comment by Githook User [ 14/Oct/15 ] |
|
Author: {u'username': u'PLW', u'name': u'Paul Pedersen', u'email': u'paul@10gen.com'}Message: |
| Comment by Githook User [ 13/Oct/15 ] |
|
Author: {u'username': u'GeertBosch', u'name': u'Geert Bosch', u'email': u'geert.bosch@mongodb.com'}Message: |
| Comment by Chad Kreimendahl [ 09/Oct/15 ] |
|
It's becoming a trend that all of the mostly highly demanded features are being moved to enterprise only. We'd all feel better if, instead of making up bogus excuses, you just came out and said "because we think it'll make you pay". We heard the nonsensical reasons for $lookup via the blog... what's the reason for in-memory to be enterprise only? |
| Comment by Ron Avnur [ 18/Sep/15 ] |
|
Tim, It will remain in enterprise builds for the foreseeable future and we will definitely provide an update if/when this changes. Ron |
| Comment by Tim Niemueller [ 17/Sep/15 ] |
|
Will this eventually be added to the open source version? |
| Comment by Ron Avnur [ 17/Sep/15 ] |
|
Ben, We are continuing to work on an in-memory storage engine and plan to release it in the MongoDB Enterprise Server, which is why you haven't seen any public commits. The in-memory storage engine is not yet available for beta testing. When it is, you'll be able to find MongoDB Enterprise Server development builds at https://www.mongodb.com/customer-evaluation-downloads-development-versions. Ron |
| Comment by Ben McCann [ 16/Sep/15 ] |
|
It seems like the in-memory storage engine already is in 3.0? See https://jira.mongodb.org/browse/DOCS-5011 |
| Comment by Ben McCann [ 16/Sep/15 ] |
|
Seems like a lot of work was done on this last year. Any chance it'll make it into 3.2 in even an experimental state? It'd be really helpful for setting up test environments. |
| Comment by guipulsar [ 22/Dec/14 ] |
|
i 'm actually facing a big difficulty to find an alternative solution to that , create in memory temp result and use it straight away with another query |
| Comment by Githook User [ 27/Oct/14 ] |
|
Author: {u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}Message: |
| Comment by Githook User [ 27/Oct/14 ] |
|
Author: {u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}Message: |
| Comment by Githook User [ 27/Oct/14 ] |
|
Author: {u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}Message:
|
| Comment by Githook User [ 27/Oct/14 ] |
|
Author: {u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}Message: |
| Comment by Eliot Horowitz (Inactive) [ 17/Sep/14 ] |
|
Author: {u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}Message: |
| Comment by Githook User [ 05/Aug/14 ] |
|
Author: {u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}Message: |
| Comment by Githook User [ 05/Aug/14 ] |
|
Author: {u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}Message: |
| Comment by Githook User [ 01/Aug/14 ] |
|
Author: {u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}Message: |
| Comment by Githook User [ 01/Aug/14 ] |
|
Author: {u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}Message: |
| Comment by Githook User [ 01/Aug/14 ] |
|
Author: {u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}Message: |
| Comment by Githook User [ 01/Aug/14 ] |
|
Author: {u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}Message: |
| Comment by Githook User [ 30/Jul/14 ] |
|
Author: {u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}Message: |
| Comment by Githook User [ 20/Jun/14 ] |
|
Author: {u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}Message: |
| Comment by Githook User [ 19/Jun/14 ] |
|
Author: {u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}Message: |
| Comment by Githook User [ 19/Jun/14 ] |
|
Author: {u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}Message: |
| Comment by Githook User [ 19/Jun/14 ] |
|
Author: {u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}Message: |
| Comment by Githook User [ 18/Jun/14 ] |
|
Author: {u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}Message: |
| Comment by Markus Haslinger [ 21/Jul/13 ] |
|
I prefer MongoDB over RavenDB (in many ways) but I'm currently forced to use RavenDB for apps in a shared environment because it has an embedded mode, so please relieve me |
| Comment by Oded Maimon [ 02/May/13 ] |
|
I've done this in the past, it does work as it should but it complicates the mongodb maintenance, it will be cleaner to have mongodb know about it and handle it by itself. BTW, this same solution can be done by DB level if you use --directoryperdb and only use tmpfs on the specific db directory. |
| Comment by Antoine Girbal [ 01/May/13 ] |
|
This should be working fine as-is, see a write up on it: |
| Comment by Phil Idem [ 08/Oct/12 ] |
|
I see that a few people have proposed using in-memory databases for the purpose of caching and testing. I would also like to suggest this feature for the purpose of temporary collections. For example, the output of a Map-Reduce job could be directed to a temporary collection. Since the collection is derived from other data there is no need to persist it to disk. The temporary collection could then be used by the application. Performance should be improved by avoiding unnecessary disk I/O. |
| Comment by Jean-Pierre Bergamin [ 28/Jun/12 ] |
|
I'd like to see an in-memory solution to simplify integration testing with MongoDB. |
| Comment by Scott Lyons [ 21/Jun/12 ] |
|
I'm going to chime in that it would be great to have this feature. Even a capped collection that was only in memory would help a lot. |
| Comment by Evgeny Myasishchev [ 04/Feb/12 ] |
|
In SQLite if you specify database name as :memory: then in-memory database is used. I love this feature of SQLite, it makes testing really fast. I really miss this feature in mongo, hope something similar will be possible some day. |
| Comment by Rakesh Madhavakurup [ 30/Jan/12 ] |
|
can this be prioritised please? |
| Comment by Alex Barnes [ 23/Dec/11 ] |
|
Being able to fire up an in-memory mongo instance to run tests against would be very useful. |
| Comment by Remon van Vliet [ 20/Apr/11 ] |
|
@auto : No, you can't just do dropDatabase because that'd still require an on-disk database which is what you want to avoid on unit test machines that may run many thousands of tests @Dominique : Not sure of LRU collections are part of this issue, but it'd be a great feature to have once MongoDB allows various ways to use hashing for shard addressing (rather than the current chunk size based balancing to distribute data) |
| Comment by Dominique De Vito [ 19/Apr/11 ] |
|
It would be useful to enable to define in-memory-only LRU collections (not only capped collections, but truly LRU collections). Then, Mongo could be used in place of memcache. |
| Comment by auto [ 28/Oct/10 ] |
|
Why can't you just do a dropDatabase before and after? |
| Comment by Remon van Vliet [ 28/Oct/10 ] |
|
We need this too. The in memory file system option is rarely actually an option. Other than performance gains (which is not that relevant to me) a more important issue is testability. Most automated testing requires a short lived mongod process that does not perform any sort of disk IO. This way we can create isolated integration tests without having to setup a mini version of production deploy environments. For example, on RDBMS systems our (and many other companies') tests look like this 1) Start in memory DB Mongo is currently blocking us on 1) which makes it a less than ideal choice for environments with this sort of testing. |
| Comment by Victor Andritoiu [ 21/Jun/10 ] |
|
Sure, it is a possibility, but is it as efficient as a native solution ? |
| Comment by phpMoAdmin [ 18/Jun/10 ] |
|
Hi Richard, your suggestion is exactly what I have posted at http://www.phpmoadmin.com/mongodb-in-memory-database - it works but there are limitations with this method. |
| Comment by Richard Kreuter (Inactive) [ 18/Jun/10 ] |
|
Some operating systems offer in-memory file system formats (e.g., Linux's tmpfs); you could probably have a mongod use an in-memory file system for the dbpath... |
| Comment by Eliot Horowitz (Inactive) [ 18/Jun/10 ] |
|
Its not that hard, so will probably happen, just a prioritization issue. |