[SERVER-1153] Create an in-memory only storage engine Created: 25/May/10  Updated: 22/Mar/17  Resolved: 09/Nov/15

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: 3.2.0-rc2

Type: New Feature Priority: Major - P3
Reporter: phpMoAdmin Assignee: Geert Bosch
Resolution: Done Votes: 101
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on WT-2178 Add ability to create in-memory only ... Closed
Related
is related to SERVER-3215 Anonymous Garbage Collected/Temporary... Closed
is related to DOCS-6802 Add documentation for in-memory stora... Closed
Tested
Backwards Compatibility: Fully Compatible
Sprint: QuInt C (11/23/15)
Participants:

 Description   

Mongo is already a performance leader, but it would be great to push this farther via an option for storage entirely within ramspace like an Oracle In-Memory Database or MySQL memory table (aka. heap table)

This would also need to turn off the disk-space pre-allocation; with storage in RAM you would not need any pre-allocation since re-allocating additional space would be an almost instant task.

I have created a hack for this that works but even with nssize=2 and smallfiles set the default disk pre-allocation makes my setup rather limited in use.
http://www.phpmoadmin.com/mongodb-in-memory-database



 Comments   
Comment by Geert Bosch [ 09/Nov/15 ]

The enterprise build contains a new in-memory storage engine for production use. It is currently in beta and is available in the enterprise build since 3.2.0-rc2. To enable, use --storageEngine inMemory. More documentation coming soon in the release notes for v3.2.

The enterprise server is openly available for free trial and development. You may use it in development / eval without a commercial license (https://www.mongodb.com/lp/download/mongodb-enterprise).

For the watchers of this ticket who are looking for a non-persistant storage engine for testing purposes, we have provided a storage engine for this use case in the community build. Use --storageEngine ephemeralForTest to enable an in-memory solution which supports collection level locking. As the name of this option suggests, do not use it in production.

Comment by Githook User [ 02/Nov/15 ]

Author:

{u'username': u'GeertBosch', u'name': u'Geert Bosch', u'email': u'geert@mongodb.com'}

Message: SERVER-1153: Add Evergreen testing for inMemory storage engine
Branch: master
https://github.com/mongodb/mongo/commit/70c5435a48b5a8e8c6559371b1136ecdca0d49ae

Comment by Githook User [ 02/Nov/15 ]

Author:

{u'username': u'GeertBosch', u'name': u'Geert Bosch', u'email': u'geert@mongodb.com'}

Message: SERVER-1153: Initial integration for in-memory storage engine based on WT
Branch: master
https://github.com/10gen/mongo-enterprise-modules/commit/880f81a1196341d8ddbb7e3253d7eb24b4be1cc9

Comment by Githook User [ 02/Nov/15 ]

Author:

{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

Message: SERVER-1153: Add and use isEphemeral predicate
Branch: master
https://github.com/mongodb/mongo/commit/ae3a6452d208f848d54f7808ac22229a228665c5

Comment by Githook User [ 16/Oct/15 ]

Author:

{u'username': u'monkey101', u'name': u'Dan Pasette', u'email': u'dan@10gen.com'}

Message: Revert "SERVER-1153: Initial commit of inMemory storage engine"

This reverts commit e0001f87181b6b195760ccd872e719a399475778.
Branch: master
https://github.com/10gen/mongo-enterprise-modules/commit/06358579119d7920d14269216f83797a5352def2

Comment by Chad Kreimendahl [ 16/Oct/15 ]

The fact that it's enterprise only, given how high it's been rated among everyone is more curious than the notifications, Tim

Comment by Tim Niemueller [ 15/Oct/15 ]

It's rather frustrating to even be bothered by notifications about commits to a non-publicly accessible repository that implements a feature that obviously many (on the open source or academic side) need – to say the least.

Comment by Githook User [ 14/Oct/15 ]

Author:

{u'username': u'PLW', u'name': u'Paul Pedersen', u'email': u'paul@10gen.com'}

Message: SERVER-1153: Initial commit of inMemory storage engine
Branch: master
https://github.com/10gen/mongo-enterprise-modules/commit/e0001f87181b6b195760ccd872e719a399475778

Comment by Githook User [ 13/Oct/15 ]

Author:

{u'username': u'GeertBosch', u'name': u'Geert Bosch', u'email': u'geert.bosch@mongodb.com'}

Message: SERVER-1153: Rename to inMemory storage engine
Branch: artree
https://github.com/10gen/mongo-enterprise-modules/commit/a688dba787f46d9f078414b3781ba50f7e31f5f4

Comment by Chad Kreimendahl [ 09/Oct/15 ]

It's becoming a trend that all of the mostly highly demanded features are being moved to enterprise only. We'd all feel better if, instead of making up bogus excuses, you just came out and said "because we think it'll make you pay".

We heard the nonsensical reasons for $lookup via the blog... what's the reason for in-memory to be enterprise only?

Comment by Ron Avnur [ 18/Sep/15 ]

Tim,

It will remain in enterprise builds for the foreseeable future and we will definitely provide an update if/when this changes.

Ron

Comment by Tim Niemueller [ 17/Sep/15 ]

Will this eventually be added to the open source version?

Comment by Ron Avnur [ 17/Sep/15 ]

Ben,

We are continuing to work on an in-memory storage engine and plan to release it in the MongoDB Enterprise Server, which is why you haven't seen any public commits.

The in-memory storage engine is not yet available for beta testing. When it is, you'll be able to find MongoDB Enterprise Server development builds at https://www.mongodb.com/customer-evaluation-downloads-development-versions.

Ron

Comment by Ben McCann [ 16/Sep/15 ]

It seems like the in-memory storage engine already is in 3.0? See https://jira.mongodb.org/browse/DOCS-5011

Comment by Ben McCann [ 16/Sep/15 ]

Seems like a lot of work was done on this last year. Any chance it'll make it into 3.2 in even an experimental state? It'd be really helpful for setting up test environments.

Comment by guipulsar [ 22/Dec/14 ]

i 'm actually facing a big difficulty to find an alternative solution to that , create in memory temp result and use it straight away with another query
could be very usefull , simulate a kind of join and solve many pb...
example in my case :
i have a query A who never result change , this query could be easely cached and so on but more,
next i could search from query A with an IN statement..
there are ton of cases where this feature is helpfull;

Comment by Githook User [ 27/Oct/14 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-1153 Create heap1_engine_test unittest
Branch: master
https://github.com/mongodb/mongo/commit/5298705477bf30ebb1cc2b95732651750f6f27f0

Comment by Githook User [ 27/Oct/14 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-1153 Heap1 is now using KVEngine
Branch: master
https://github.com/mongodb/mongo/commit/6d7f968a9495b504bea981e392f5ec60a947123e

Comment by Githook User [ 27/Oct/14 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-1153 Improve RecordStoreHeap

Comment by Githook User [ 27/Oct/14 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-1153 Clean up Heap1RecoveryUnit and implement recordChange()
Branch: master
https://github.com/mongodb/mongo/commit/1ccde1e095c8b18280417fd2ef638db7e956423a

Comment by Eliot Horowitz (Inactive) [ 17/Sep/14 ]

Author:

{u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-1152: make SortedDataInterface work for heap1
Branch: master
https://github.com/mongodb/mongo/commit/cce6a04bcb073699a9e16a6ce9eca587060f8c3f

Comment by Githook User [ 05/Aug/14 ]

Author:

{u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-1153: hack index linking to make unit testing heap possible
Branch: master
https://github.com/mongodb/mongo/commit/6499f50b1fb6333096b8c358deededb920738f7e

Comment by Githook User [ 05/Aug/14 ]

Author:

{u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-1153: clean max handling in heap1 capped collection
Branch: master
https://github.com/mongodb/mongo/commit/5d311d9f0583e3d74ce1aa47c66965cd0b39276f

Comment by Githook User [ 01/Aug/14 ]

Author:

{u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-13635 SERVER-1153: fix collmod test to be genric and support in heap1
Branch: master
https://github.com/mongodb/mongo/commit/a3e39c37f750fe3b933a970a470a96780b71e54b

Comment by Githook User [ 01/Aug/14 ]

Author:

{u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-1153: fix heap1 handling of large keys
Branch: master
https://github.com/mongodb/mongo/commit/c869fa7ca774be7178d540ba023dad42be2a169a

Comment by Githook User [ 01/Aug/14 ]

Author:

{u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-1153: implement base of HeapRecordStore::setCustomOption
Branch: master
https://github.com/mongodb/mongo/commit/d7a94e259c94a46feab8fabebbd8c4dc0eebce8f

Comment by Githook User [ 01/Aug/14 ]

Author:

{u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-1153: fix reverseIterator for heap record store
Branch: master
https://github.com/mongodb/mongo/commit/93625eb6a773f0ecdd0324afba4926f35c9abe43

Comment by Githook User [ 30/Jul/14 ]

Author:

{u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-1153: fix bug in save/restore cursor handling for heap1
Branch: master
https://github.com/mongodb/mongo/commit/7b3a7003634778690cf3114d30b16b0f49a01711

Comment by Githook User [ 20/Jun/14 ]

Author:

{u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-1153: head record store update now works
Branch: master
https://github.com/mongodb/mongo/commit/152dd4f70ec3c627b7ceec6ff52855ab6c3cb341

Comment by Githook User [ 19/Jun/14 ]

Author:

{u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-1153: turn off unit test until can clean library dependencies
Branch: master
https://github.com/mongodb/mongo/commit/77472f85d6cadf99cece2019063dd35067ff682f

Comment by Githook User [ 19/Jun/14 ]

Author:

{u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-1153: insert and find works
Branch: master
https://github.com/mongodb/mongo/commit/5eae3ef25468bba0a85640ddd578326c7381d596

Comment by Githook User [ 19/Jun/14 ]

Author:

{u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-13635 SERVER-1153: can choose heap1 at startup, but doesn't remotely work yet
Branch: master
https://github.com/mongodb/mongo/commit/b4cafb46595d6e889f8effdca16e98c3f505078e

Comment by Githook User [ 18/Jun/14 ]

Author:

{u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-1153: some very simple db ops work
Branch: master
https://github.com/mongodb/mongo/commit/07a28c5089142100c15ca403eabdfc535e03a229

Comment by Markus Haslinger [ 21/Jul/13 ]

I prefer MongoDB over RavenDB (in many ways) but I'm currently forced to use RavenDB for apps in a shared environment because it has an embedded mode, so please relieve me

Comment by Oded Maimon [ 02/May/13 ]

I've done this in the past, it does work as it should but it complicates the mongodb maintenance, it will be cleaner to have mongodb know about it and handle it by itself.

BTW, this same solution can be done by DB level if you use --directoryperdb and only use tmpfs on the specific db directory.

Comment by Antoine Girbal [ 01/May/13 ]

This should be working fine as-is, see a write up on it:
http://edgystuff.tumblr.com/post/49304254688/how-to-use-mongodb-as-a-pure-in-memory-db-redis-style

Comment by Phil Idem [ 08/Oct/12 ]

I see that a few people have proposed using in-memory databases for the purpose of caching and testing. I would also like to suggest this feature for the purpose of temporary collections. For example, the output of a Map-Reduce job could be directed to a temporary collection. Since the collection is derived from other data there is no need to persist it to disk. The temporary collection could then be used by the application. Performance should be improved by avoiding unnecessary disk I/O.

Comment by Jean-Pierre Bergamin [ 28/Jun/12 ]

I'd like to see an in-memory solution to simplify integration testing with MongoDB.

Comment by Scott Lyons [ 21/Jun/12 ]

I'm going to chime in that it would be great to have this feature. Even a capped collection that was only in memory would help a lot.

Comment by Evgeny Myasishchev [ 04/Feb/12 ]

In SQLite if you specify database name as :memory: then in-memory database is used. I love this feature of SQLite, it makes testing really fast. I really miss this feature in mongo, hope something similar will be possible some day.

Comment by Rakesh Madhavakurup [ 30/Jan/12 ]

can this be prioritised please?

Comment by Alex Barnes [ 23/Dec/11 ]

Being able to fire up an in-memory mongo instance to run tests against would be very useful.

Comment by Remon van Vliet [ 20/Apr/11 ]

@auto : No, you can't just do dropDatabase because that'd still require an on-disk database which is what you want to avoid on unit test machines that may run many thousands of tests

@Dominique : Not sure of LRU collections are part of this issue, but it'd be a great feature to have once MongoDB allows various ways to use hashing for shard addressing (rather than the current chunk size based balancing to distribute data)

Comment by Dominique De Vito [ 19/Apr/11 ]

It would be useful to enable to define in-memory-only LRU collections (not only capped collections, but truly LRU collections).

Then, Mongo could be used in place of memcache.
Users may be then able to use Mongo into different tiers (as a cache, into the middle tiers, and/or as back-end storage into the last tiers).

Comment by auto [ 28/Oct/10 ]

Why can't you just do a dropDatabase before and after?

Comment by Remon van Vliet [ 28/Oct/10 ]

We need this too. The in memory file system option is rarely actually an option. Other than performance gains (which is not that relevant to me) a more important issue is testability. Most automated testing requires a short lived mongod process that does not perform any sort of disk IO. This way we can create isolated integration tests without having to setup a mini version of production deploy environments.

For example, on RDBMS systems our (and many other companies') tests look like this

1) Start in memory DB
2) Import database start state (e.g. .sql import)
3) Run tested component code
4) Validate results
5) Pass/fail the test in build system reporting

Mongo is currently blocking us on 1) which makes it a less than ideal choice for environments with this sort of testing.

Comment by Victor Andritoiu [ 21/Jun/10 ]

Sure, it is a possibility, but is it as efficient as a native solution ?
I am not sure that MongoDB could cover my needs, even in this situation, but from the conceptual point of view, it seems to be the best way to do.
In terms of performances, I need to be able to update about 10000 parameters (let's say integers) within 50ms on something like 10 processing units (standard PC based on Core2Duo in the best case and Atom in the worst) using an Ethernet Gigabyte network (in a lab, means short distances <5m of cable length, full duplex etc, so the network performances are not a problem, I think).
Any way if I can find voters...
And thanks for all the comments.

Comment by phpMoAdmin [ 18/Jun/10 ]

Hi Richard, your suggestion is exactly what I have posted at http://www.phpmoadmin.com/mongodb-in-memory-database - it works but there are limitations with this method.

Comment by Richard Kreuter (Inactive) [ 18/Jun/10 ]

Some operating systems offer in-memory file system formats (e.g., Linux's tmpfs); you could probably have a mongod use an in-memory file system for the dbpath...

Comment by Eliot Horowitz (Inactive) [ 18/Jun/10 ]

Its not that hard, so will probably happen, just a prioritization issue.
The more votes you can drum up for the case the better

Generated at Thu Feb 08 02:56:13 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.