[SERVER-4107] Server exits after JS Error: out of memory nofile_b:7 Created: 20/Oct/11  Updated: 29/Feb/12  Resolved: 02/Dec/11

Status: Closed
Project: Core Server
Component/s: JavaScript
Affects Version/s: 1.8.1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Steve Fram Assignee: Antoine Girbal
Resolution: Duplicate Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ubuntu 10.4


Issue Links:
Depends
depends on SERVER-3012 JS "out of memory" should recover mor... Closed
depends on SERVER-4124 try to force full js engine reset aft... Closed
depends on SERVER-4414 add option to choose the JS memory size Closed
Operating System: ALL
Participants:

 Description   

Our production server, which is under moderate load occasionally shows the javascript error shown in the attached gist and exits (of course, bringing our entire site down, paging, etc.).

https://gist.github.com/5180b96319f3c4453d58

We don't have any custom javascript running, nor are the queries typically very complex.



 Comments   
Comment by Scott Hernandez (Inactive) [ 16/Dec/11 ]

Since the javascript engine is compiled in if there is a memory issue
then it causes the shutdown. There isn't much we can do about that. If
there is invalid memory access from the JavaScript engine it could
cause errors anywhere in the running program so once it happens
nothing is safe anymore.

Comment by Antoine Girbal [ 02/Dec/11 ]

This issue will be resolved by implementing other issues depended on. Note that the likely culprit is SERVER-3538 which is fixed in upcoming 2.0.2

Comment by Eliot Horowitz (Inactive) [ 23/Oct/11 ]

Making the memory limit tunable might be an option, but it would just buy more time, not fix the underlying issue.
Process isolation is also an option, but would reduce the performance somewhat significantly in many cases.

We're actually talking to the v8 team about possible options.

Comment by Chris Eppstein [ 23/Oct/11 ]

@eliot glad to hear you are working on it. If there's no good solution at this time why isn't the memory limit tunable? Also it seems like you could manage this via process isolation like they do in chrome.

Comment by Eliot Horowitz (Inactive) [ 22/Oct/11 ]

The fixes antoine mentions mitigate a large number of issues - but the js engines still get very unstable once they run out of memory.
We're testing v8 to see if its better than spidermonkey in this regard, but hasn't been very promising.

Comment by Eliot Horowitz (Inactive) [ 21/Oct/11 ]

@chris - To clarify, the error condition is not acceptable, and we are working on solutions. We are issues in the js implementations we use, so its hard to have a specific ETA on a fix.

Comment by Chris Eppstein [ 21/Oct/11 ]

We only use these queries as part of nightly jobs that update our documents – they are not done as part of the request/response lifecycle. So these are not our most common queries – they exist to optimize our most common queries.

Are you saying that if I used inline javascript instead of an uploaded javascript, this failure condition wouldn't exist?

We will upgrade, but when I see your CTO say that this fundamental instability as a result normal operations is reasonable it really worries me. I hope that you will retract or clarify your statement that crashing during a query is ever a reasonable behavior for critical infrastructure. We are using MongoDB in this part of our product as a test with an eye towards rolling it out to more of our product and I've been telling other people who trust me that they should use it. I am now re-thinking my position.

Comment by Antoine Girbal [ 21/Oct/11 ]

a few things that are related to this ticket:

  • as much as possible, it is better to avoid javascript execution for very frequent querying.
    MongoDB is a C++ db and only gives 8MB of ram for JS to execute in 1.8.1.
    Basically if a query can be expressed in both JS and query language, avoid JS.
    Note that the JS heap was increased to 64MB in 1.8.3 and 2.0.
  • SERVER-3528: we discovered that the js scope in spidermonkey keeps increasing over time.
    We now make sure they get recycled.
  • SERVER-3012: with this ticket we make sure we clear the scope cache in case of an out of memory.
    This way if a single query goes to use over the memory limit, it will not affect the js engine further.
Comment by Chris Eppstein [ 21/Oct/11 ]

It is invoked like this:

$where: "resourceNotInRegion(this, 'ZipCode', 'region-name', '123456')"

Comment by Eliot Horowitz (Inactive) [ 21/Oct/11 ]

There should be more than that.
A $where function has "this" set to the current doc

Comment by Chris Eppstein [ 21/Oct/11 ]

it's used in a $where clause in some queries.

Comment by Eliot Horowitz (Inactive) [ 21/Oct/11 ]

How are you invoking that, etc.?
Doesn't seem it can be used directly.

Comment by Chris Eppstein [ 21/Oct/11 ]

I have updated the gist to include the js file that is running in our system

Comment by Eliot Horowitz (Inactive) [ 21/Oct/11 ]

Can you sen all the ja code you use? So $where db.eval and map/reduce? Nothing else usies js

Comment by Chris Eppstein [ 21/Oct/11 ]

Who is @paul?

WTF guys. this is a javascript based database and you're telling me that it's unstable even at 2.0? I'm crying. Please tell me what we can do to avoid this.

Comment by Eliot Horowitz (Inactive) [ 21/Oct/11 ]

@chris - we are trying to make the js engines more robust ( SERVER-3012 ) but at current they can get very unstable after out of memory errors

Comment by Eliot Horowitz (Inactive) [ 21/Oct/11 ]

@paul - can you also send the $where code and any other js code you use?

Comment by Eliot Horowitz (Inactive) [ 21/Oct/11 ]

This isn't a single query failure.
The query (because of a bug or something like that) creates an invalid internal state which is not recoverable from.
All subsequent js queries would fail.

Comment by Chris Eppstein [ 21/Oct/11 ]

No. Shutting down is not at all reasonable. This query should error out but the database should keep working. We are simply using basic, documented features of mongodb and those should never be able to crash the whole db. Does PLSQL crash mysql? No. Frankly, I'm astonished by your answer.

Comment by Eliot Horowitz (Inactive) [ 21/Oct/11 ]

This error indicates a catastrophic js engine failure, so shutting down is reasonable.

Comment by Steve Fram [ 21/Oct/11 ]

Thank you. will try some upgrades.

Question – do you think it is the appropriate behavior for the entire server to shut down in cases where a single query fails ? I certainly would rather that this memory management problem trigger an error state, rather than causing a catastrophic failure.

Comment by Scott Hernandez (Inactive) [ 20/Oct/11 ]

It looks like you are using $where (javascript) in your queries. The javascript heap size has been increased in 2.0.x and should help reduce these kinds of issues (memory allocation for js).

Please think about upgrading to 2.0.1 when it comes out in the next few days.

We would also suggest upgrading to 1.8.3/4 as there have been many bug fixes from since 1.8.1.

Generated at Thu Feb 08 03:04:59 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.