[SERVER-4107] Server exits after JS Error: out of memory nofile_b:7 Created: 20/Oct/11 Updated: 29/Feb/12 Resolved: 02/Dec/11 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | JavaScript |
| Affects Version/s: | 1.8.1 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Steve Fram | Assignee: | Antoine Girbal |
| Resolution: | Duplicate | Votes: | 1 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
ubuntu 10.4 |
||
| Issue Links: |
|
||||||||||||||||
| Operating System: | ALL | ||||||||||||||||
| Participants: | |||||||||||||||||
| Description |
|
Our production server, which is under moderate load occasionally shows the javascript error shown in the attached gist and exits (of course, bringing our entire site down, paging, etc.). https://gist.github.com/5180b96319f3c4453d58 We don't have any custom javascript running, nor are the queries typically very complex. |
| Comments |
| Comment by Scott Hernandez (Inactive) [ 16/Dec/11 ] |
|
Since the javascript engine is compiled in if there is a memory issue |
| Comment by Antoine Girbal [ 02/Dec/11 ] |
|
This issue will be resolved by implementing other issues depended on. Note that the likely culprit is |
| Comment by Eliot Horowitz (Inactive) [ 23/Oct/11 ] |
|
Making the memory limit tunable might be an option, but it would just buy more time, not fix the underlying issue. We're actually talking to the v8 team about possible options. |
| Comment by Chris Eppstein [ 23/Oct/11 ] |
|
@eliot glad to hear you are working on it. If there's no good solution at this time why isn't the memory limit tunable? Also it seems like you could manage this via process isolation like they do in chrome. |
| Comment by Eliot Horowitz (Inactive) [ 22/Oct/11 ] |
|
The fixes antoine mentions mitigate a large number of issues - but the js engines still get very unstable once they run out of memory. |
| Comment by Eliot Horowitz (Inactive) [ 21/Oct/11 ] |
|
@chris - To clarify, the error condition is not acceptable, and we are working on solutions. We are issues in the js implementations we use, so its hard to have a specific ETA on a fix. |
| Comment by Chris Eppstein [ 21/Oct/11 ] |
|
We only use these queries as part of nightly jobs that update our documents – they are not done as part of the request/response lifecycle. So these are not our most common queries – they exist to optimize our most common queries. Are you saying that if I used inline javascript instead of an uploaded javascript, this failure condition wouldn't exist? We will upgrade, but when I see your CTO say that this fundamental instability as a result normal operations is reasonable it really worries me. I hope that you will retract or clarify your statement that crashing during a query is ever a reasonable behavior for critical infrastructure. We are using MongoDB in this part of our product as a test with an eye towards rolling it out to more of our product and I've been telling other people who trust me that they should use it. I am now re-thinking my position. |
| Comment by Antoine Girbal [ 21/Oct/11 ] |
|
a few things that are related to this ticket:
|
| Comment by Chris Eppstein [ 21/Oct/11 ] |
|
It is invoked like this: $where: "resourceNotInRegion(this, 'ZipCode', 'region-name', '123456')" |
| Comment by Eliot Horowitz (Inactive) [ 21/Oct/11 ] |
|
There should be more than that. |
| Comment by Chris Eppstein [ 21/Oct/11 ] |
|
it's used in a $where clause in some queries. |
| Comment by Eliot Horowitz (Inactive) [ 21/Oct/11 ] |
|
How are you invoking that, etc.? |
| Comment by Chris Eppstein [ 21/Oct/11 ] |
|
I have updated the gist to include the js file that is running in our system |
| Comment by Eliot Horowitz (Inactive) [ 21/Oct/11 ] |
|
Can you sen all the ja code you use? So $where db.eval and map/reduce? Nothing else usies js |
| Comment by Chris Eppstein [ 21/Oct/11 ] |
|
Who is @paul? WTF guys. this is a javascript based database and you're telling me that it's unstable even at 2.0? I'm crying. Please tell me what we can do to avoid this. |
| Comment by Eliot Horowitz (Inactive) [ 21/Oct/11 ] |
|
@chris - we are trying to make the js engines more robust ( |
| Comment by Eliot Horowitz (Inactive) [ 21/Oct/11 ] |
|
@paul - can you also send the $where code and any other js code you use? |
| Comment by Eliot Horowitz (Inactive) [ 21/Oct/11 ] |
|
This isn't a single query failure. |
| Comment by Chris Eppstein [ 21/Oct/11 ] |
|
No. Shutting down is not at all reasonable. This query should error out but the database should keep working. We are simply using basic, documented features of mongodb and those should never be able to crash the whole db. Does PLSQL crash mysql? No. Frankly, I'm astonished by your answer. |
| Comment by Eliot Horowitz (Inactive) [ 21/Oct/11 ] |
|
This error indicates a catastrophic js engine failure, so shutting down is reasonable. |
| Comment by Steve Fram [ 21/Oct/11 ] |
|
Thank you. will try some upgrades. Question – do you think it is the appropriate behavior for the entire server to shut down in cases where a single query fails ? I certainly would rather that this memory management problem trigger an error state, rather than causing a catastrophic failure. |
| Comment by Scott Hernandez (Inactive) [ 20/Oct/11 ] |
|
It looks like you are using $where (javascript) in your queries. The javascript heap size has been increased in 2.0.x and should help reduce these kinds of issues (memory allocation for js). Please think about upgrading to 2.0.1 when it comes out in the next few days. We would also suggest upgrading to 1.8.3/4 as there have been many bug fixes from since 1.8.1. |