[SERVER-3012] JS "out of memory" should recover more gracefully Created: 28/Apr/11  Updated: 12/Jul/16  Resolved: 06/Mar/12

Status: Closed
Project: Core Server
Component/s: JavaScript
Affects Version/s: None
Fix Version/s: 2.1.1

Type: Bug Priority: Major - P3
Reporter: Kyle Banker Assignee: Antoine Girbal
Resolution: Done Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-3656 increase spidermonkey JS heap limit t... Closed
is depended on by SERVER-4107 Server exits after JS Error: out of m... Closed
Operating System: ALL
Participants:

 Description   

Once you get an "out of memory" error, the JavaScript runtime becomes unusable.

// Add this to system.js
{ "id" : "f1", "value" : function cf2f {
a = [];
b = [];
c = [];
for (i = 0; i < n; i++)

{ a.push(Math.random()); b.push(Math.random()); c.push(Math.random()); }

} }

// Then execute this:
> db.eval("f1(10)")
null
> db.eval("f1(10000)")
null
> db.eval("f1(100000)")
null
> db.eval("f1(1000000)")
Thu Apr 28 11:21:45 uncaught exception: {
"errno" : -3,
"errmsg" : "invoke failed: JS Error: out of memory nofile_b:6",
"ok" : 0
}
> db.eval("f1(10)")
Thu Apr 28 11:21:49 uncaught exception: {
"errno" : -3,
"errmsg" : "invoke failed: JS Error: out of memory nofile_b:1",
"ok" : 0
}



 Comments   
Comment by auto [ 06/Mar/12 ]

Author:

{u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}

Message: SERVER-3012: added $where tests
Branch: master
https://github.com/mongodb/mongo/commit/276e5f09ec00fdc7b8131664f4587fe6e9094e74

Comment by auto [ 06/Mar/12 ]

Author:

{u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}

Message: SERVER-3012: beefed up test
Branch: master
https://github.com/mongodb/mongo/commit/0e5a0113febadbb8a5e7f9a7b7e43e479c0b0640

Comment by Antoine Girbal [ 06/Mar/12 ]

behavior is now correct on both SM and V8.
A call to JS on server that uses more than 64MB will get OOM exception.
After that the engine is still usable for new calls.

Comment by auto [ 06/Mar/12 ]

Author:

{u'login': u'agirbal', u'email': u'antoine@10gen.com', u'name': u'agirbal'}

Message: SERVER-3012: proper out of memory handling for v8
Branch: master
https://github.com/mongodb/mongo/commit/42addfe2e5a49d730b5cb806b7bc9dcc4becc12b

Comment by auto [ 13/Oct/11 ]

Author:

{u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}

Message: - SERVER-3012: cant make V8 go out of mem, it will affect further tests since smoke.py uses common instance
Branch: master
https://github.com/mongodb/mongo/commit/c6891bbb029a6bf7e39d5dfa1883cb72ad1fd217

Comment by auto [ 07/Oct/11 ]

Author:

{u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}

Message: - SERVER-3012: added test
Branch: master
https://github.com/mongodb/mongo/commit/2ed73366ebd945895fe1b3882f4978a866722259

Comment by auto [ 29/Sep/11 ]

Author:

{u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}

Message: - SERVER-3972: Add memory limits to V8 JS

Comment by Antoine Girbal [ 29/Sep/11 ]

V8 turns out to be a pain.
By default an OOM throws a fatal error and makes the engine unusable.
There is a method called IgnoreOutOfMemoryException which makes the error not throw.
But as soon as another scope is obtained, it gets an error like:

#

  1. Fatal error in src/handles-inl.h, line 64
  2. CHECK(location_ != __null) failed
    #

Even after throwing out all contexts and calling GC.
I checked the IsDead() and it returns false..

A thread states that it is very unreliable to try to recover from out of mem
http://markmail.org/message/p5ofyegao3pugubz#query:+page:1+mid:eng2hmpzllcgunrc+state:results
It suggests to use Isolates to obtain a new V8 environment.
Since we want to use Isolates for multi-threading, this may be fixed then.

Comment by Antoine Girbal [ 26/Sep/11 ]

code will now destroy JS context and any idle contexts.
Added test.
Need to make it work with V8 too.

Comment by Eliot Horowitz (Inactive) [ 26/Sep/11 ]

Once there is a GC - destroying that context would make a lot of sense.

Comment by Antoine Girbal [ 26/Sep/11 ]

We cant do too much to help if someone is using over the mem limit by setting global variables.
We cant really go around unsetting variables.
the js engine will try to GC as much as it can.
The one thing we could do is empty the js context pool to make a bit of room.

Comment by Antoine Girbal [ 26/Sep/11 ]

The reason why the engine seems unusable is that the variable a/b/c in your function are global and so they stay around even after the out of mem.
If instead you make them local using "var", they get properly cleaned up and engine is usable after OOM.

PRIMARY> db.system.js.save({ "_id" : "f1", "value" : function {
... var a = [];
... var b = [];
... var c = [];
... for (i = 0; i < n; i++)

{ a.push(Math.random()); b.push(Math.random()); c.push(Math.random()); }

... } }
... )
PRIMARY> db.eval("f1(10000000)")
Sun Sep 25 23:36:27 uncaught exception: {
"errno" : -3,
"errmsg" : "invoke failed: JS Error: out of memory nofile_b:5",
"ok" : 0
}
PRIMARY> db.eval("f1(10)")
null
PRIMARY> db.eval("f1(10000000)")
Sun Sep 25 23:36:44 uncaught exception: {
"errno" : -3,
"errmsg" : "invoke failed: JS Error: out of memory nofile_b:5",
"ok" : 0
}
PRIMARY> db.eval("f1(10)")
null

Comment by auto [ 15/Aug/11 ]

Author:

{u'login': u'agirbal', u'name': u'Antoine Girbal', u'email': u'antoine@10gen.com'}

Message: - SERVER-3012: increased SM runtime limit to 64MB
Branch: v1.8
https://github.com/mongodb/mongo/commit/d6d4ee256edd8989a74fd6716841780696154da5

Comment by auto [ 24/May/11 ]

Author:

{u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}

Message: SERVER-3012: increased SM runtime limit to 64MB
Branch: master
https://github.com/mongodb/mongo/commit/6bfc2d5d5ff748bf89f239ce0914052940e4998b

Comment by Paul Harvey [ 24/May/11 ]

Cool, I will try this ASAP.

I wonder if it would be possible to add some -vvvv[..] debug output which could help us track down possible problems in our JS. I have no idea at all how SM (or other JS engines) work their memory management, but perhaps seeing messages about GC activity or heap/stack size increases would be helpful...

I only ask because, we've already reviewed our JS (I think Sven is still going over it), and we've hammered this thing in testing and can't reproduce the fault until it's put into production. So I'm afraid we need some production runtime info to get this debugged..

Comment by Antoine Girbal [ 24/May/11 ]

Note that JS has not become unusable in my tests.
Even after OOM I am able to query again OK.
Otherwise this would point to a mem leak issue..

I think we should increase SM limit quite a bit because many ppl have reported OOM with JS.
It is too easy to hit the 8MB limit.
Even if the root cause is a mem leak, at least it would buy time.

Comment by Antoine Girbal [ 24/May/11 ]

thanks for the easily reproducible issue.
So the reason is that we have a hard limit on spidermonkey set to 8MB.
This is highly conservative I think, considering that there is only 1 sm engine and servers usually have GBs of RAM.

In engine_spidermonkey.cpp:
_runtime = JS_NewRuntime(8L * 1024L * 1024L);
can be set to
_runtime = JS_NewRuntime(128L * 1024L * 1024L);

Then your eval can go way further
> db.eval("f1(1000000)")
null
> db.eval("f1(5000000)")
null
> db.eval("f1(10000000)")
Mon May 23 23:43:32 exec error: shell/db.js:396 {
"errno" : -3,
"errmsg" : "invoke failed: JS Error: out of memory nofile_b:0",
"ok" : 0
}
throw tojson( res );
^

> db.eval("f1(1000000)")
null

Comment by Paul Harvey [ 23/May/11 ]

I filed SERVER-3131, where we are getting getting JS "out of memory" errors, followed by segfaults an hour or two later.

Generated at Thu Feb 08 03:01:49 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.