[SERVER-9051] v8 heap allocation failure can lead to segfault Created: 21/Mar/13  Updated: 14/Apr/16  Resolved: 24/Sep/15

Status: Closed
Project: Core Server
Component/s: JavaScript
Affects Version/s: 2.4.0
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Ben Becker Assignee: Unassigned
Resolution: Done Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Duplicate
is duplicated by SERVER-9117 Mongo segfault on mapreduce - repeata... Closed
Related
related to SERVER-9213 Resource constraints cause premature ... Closed
related to SERVER-9267 Issues with readonly BSON in v8 Closed
related to SERVER-9185 Add GC Prologue and Epilogue heap stats Closed
related to SERVER-9187 Reduce memory requirements from v8 in... Closed
is related to SERVER-9291 V8Scope::invoke shouldn't silently ig... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:

 Description   

The following functions may attempt to dereference an empty handle when v8 heap space is nearly depleted:

  • namedGet()
  • namedGetRO()
  • indexedGet()
  • indexedGetRO()
  • nativeCallback()

This is because mongoToV8Element() does not check for allocation success nor OOM when creating a new JS object.



 Comments   
Comment by Mira Carey [ 24/Sep/15 ]

No longer relevant as we've ported forward to SpiderMonkey, which doesn't have the underlying issue (all allocations crash in our allocator, or fail sensibly if initiated by SM)

Comment by Francois Rigault [ 09/Aug/13 ]

I got a similar problem in MongoDB v2.4.5 or v2.5.2-pre- (git version: bac3b67d20128e03487680b8d713195a18315d6e)

mongod server process core with segfault:

On server side:

use test
for (var i =0; i < 500000; i++) {
	var a = Object();
	for (var j = 0; j < 1000; j++) {
		a[j]  = "foobar";
	}	
	db.test.insert({ 'abc ' : a});
}

On client side (I'm connecting using PyMongo as mongo shell seems fine ):

from pymongo import MongoClient
connection = MongoClient('mymachine', 27017)
db = connection.test
crules = db.test
rulesBweight = crules.group([''],None,{'list': []},'function(obj, prev) {prev.list.push(obj);prev.list.push(prev)}')

mongod segfault in mongo::V8Scope::v8ToMongoObject

EDIT: still reproducable today with 2.5.4

Comment by Ben Becker [ 29/Mar/13 ]

The linked tickets resolve an issue with excessive memory consumption, which is what provoked this ticket. They have already been backported to the v2.4 branch, and should land in v2.4.2.

I'm leaving this ticket open as we still need to find a way to handle OOM conditions more gracefully.

Comment by Andy Schwerin [ 25/Mar/13 ]

We lack a facility for it now, but what you'd like in this scenario is to terminate the current operation, block further JS execution, and do a clean server shutdown. More broadly, you'd like to stop running JS inside the server process, so that the JS engine wouldn't have to share fate with the rest of the process, but that's a much larger redesign. fassertFailed() might be the only real choice for us, in the very short term. That might be the right fix for the proximate bug, but the larger issue of why V8 is reporting out-of-memory in the field needs to be addressed, too.

Comment by Ben Becker [ 25/Mar/13 ]

In debug mode, the stack trace may look like this:

#
# Fatal error in CALL_AND_RETRY_2
# Allocation failed - process out of memory
#
 
Sun Mar 24 15:51:25.389 Got signal: 6 (Abort trap: 6).
 
Sun Mar 24 15:51:25.416 Backtrace:
0x110414a7b 0x10ffd8c71 0x7fff8a57194a 0 0x7fff8a5c8dce 0x1106993fb 0x110532886 0x1104f057b 0x11050d6d2 0x11058cd42 0x110510a97 0x1103e0eb1 0x1106719ac 0x110671412 0x1106b7702 0x19a80d406362 
 0   mongod                              0x0000000110414a7b _ZN5mongo15printStackTraceERSo + 43
 1   mongod                              0x000000010ffd8c71 _ZN5mongo10abruptQuitEi + 225
 2   libsystem_c.dylib                   0x00007fff8a57194a _sigtramp + 26
 3   ???                                 0x0000000000000000 0x0 + 0
 4   libsystem_c.dylib                   0x00007fff8a5c8dce abort + 143
 5   mongod                              0x00000001106993fb _ZN2v88internal2OS5AbortEv + 9
 6   mongod                              0x0000000110532886 _Z9API_FatalPKcS0_z + 166
 7   mongod                              0x00000001104f057b _ZN2v8L24DefaultFatalErrorHandlerEPKcS1_ + 107
 8   mongod                              0x000000011050d6d2 _ZN2v88internal2V823FatalProcessOutOfMemoryEPKcb + 518
 9   mongod                              0x000000011058cd42 _ZN2v88internal16ForceSetPropertyENS0_6HandleINS0_8JSObjectEEENS1_INS0_6ObjectEEES5_18PropertyAttributes + 578
 10  mongod                              0x0000000110510a97 _ZN2v86Object8ForceSetENS_6HandleINS_5ValueEEES3_NS_17PropertyAttributeE + 283
 11  mongod                              0x00000001103e0eb1 _ZN5mongoL8namedGetEN2v85LocalINS0_6StringEEERKNS0_12AccessorInfoE + 609
 12  mongod                              0x00000001106719ac _ZN2v88internal8JSObject35GetPropertyAttributeWithInterceptorEPS1_PNS0_6StringEb + 822
 13  mongod                              0x0000000110671412 _ZN2v88internal10JSReceiver32GetPropertyAttributeWithReceiverEPS1_PNS0_6StringE + 252
 14  mongod                              0x00000001106b7702 _ZN2v88internal19Runtime_HasPropertyENS0_9ArgumentsEPNS0_7IsolateE + 114
 15  ???                                 0x000019a80d406362 0x0 + 28209567523682

Note that we're setting IgnoreOutOfMemoryException(), but it isn't checked when ForceSetProperty()'s CALL_HEAP_FUNCTION/CALL_AND_RETRY macro checks for an OOM condition.

Not sure how useful this would be, but v8::SetFatalErrorHandler() allows us to set a callback when a fatal error has been encountered. v8 will not be useable at this point, and I'm not aware of any way to cleanly free all resources at this point.

Comment by Ben Becker [ 25/Mar/13 ]

It's the v8-specific heap allocator; not a malloc failure.

Comment by Andy Schwerin [ 25/Mar/13 ]

Specifically, what allocator is returning NULL? Is it a V8-specific allocator, or is malloc failing?

Generated at Thu Feb 08 03:19:13 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.