[SERVER-20027] MongoDB 3.0.3 Segfault after jscmd error Created: 19/Aug/15  Updated: 29/Jan/16  Resolved: 25/Aug/15

Status: Closed
Project: Core Server
Component/s: JavaScript, Stability
Affects Version/s: 3.0.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Christopher Neill Assignee: Sam Kleinman (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

2015-08-18T15:03:53+00:00 rs0db2 mongod.27018[66032]: [ID 702911 local7.error] [conn32213] ReferenceError: _$jscmd is not defined#012    at _funcs9 (_funcs9:2:17) near 'if (_$jscmd("api/models/concerns/se'  (line 2)#012
2015-08-18T15:03:53+00:00 rs0db2 mongod.27018[66032]: [ID 702911 local7.crit] [conn32213] Invalid access at address: 0#012
2015-08-18T15:03:53+00:00 rs0db2 mongod.27018[66032]: [ID 702911 local7.crit] [conn32213] Got signal: 11 (Segmentation Fault).#012#012 0x14b5ef2 0x14b55a3 0x14b5cfe 0xfffffd7fff165176 0xfffffd7fff157f2b 0x1548fe0 0x1403454 0x14038d2 0x13fa4b1 0x13fa4e1 0xa807d9 0x13f334c 0x13f3de1 0xcae6af 0xcb2cb9 0xd4538c 0xd464df 0x
d4733e 0xfca785 0xe83b54 0xa8555e 0x14571f1 0xfffffd7fef354816 0xfffffd7fff164d9a 0xfffffd7fff1650b0#012----- BEGIN BACKTRACE -----#012{"backtrace":[{"b":"400000","o":"10B5EF2"},{"b":"400000","o":"10B55A3"},{"b":"400000","o":"10B5CFE"},{"b":"FFFFFD7FFF050000","o":"115176"},{"b":"FFFFFD7FFF050000","o":"107F2B"},{"b":"400000","o":"1148FE0"},{"b":"400000","o":"1003454"},{"b":"400000","o":"10038D2"},{"b":"400000","o":"FFA4B1"},{"b":"400000","o":"FFA4E1"},{"b":"400000","o":"6807D9"},{"b":"400000","o":"FF334C"},{"b":"400000","o":"FF3DE1"},{"b":"400000","o":"8AE6AF"},{"b":"400000","o":"8B2CB9"},{"b":"400000","o":"94538C"},{"b":"400000","o"
:"9464DF"},{"b":"400000","o":"94733E"},{"b":"400000","o":"BCA785"},{"b":"400000"," [...]

SunOS rs0db1.us-east-2 5.11 joyent_20150617T074149Z i86pc i386 i86pc Solaris



 Comments   
Comment by Ramon Fernandez Marina [ 29/Jan/16 ]

For the record, SERVER-22334 shows how a missing "var" keyword in the JS code could trigger this issue. MongoDB 3.2 uses SpiderMonkey as the JavaScript engine an it handles this case better than V8.

Comment by Nick Wisse [X] [ 26/Aug/15 ]

Thanks Sam

On Wed, Aug 26, 2015 at 5:29 PM, Sam Kleinman (JIRA) <jira@mongodb.org>

{ name : "Nick Wisse", title : "Corporate Account Executive", phone : "+1 (866) 237-8815 Ext. 7143", email : "nick.wisse@MongoDB.com <nick.wisse@10gen.com>[image: Look up in Salesforce]", location : "New York, NY", twitter : ["@NickMongoDB <https://twitter.com/NickMongoDB>", "@M <https://twitter.com/mongodb>ongoDBinc"], facebook : ["MongoDB <https://www.facebook.com/mongodb>"] }

Hot New News:

Comment by Christopher Neill [ 26/Aug/15 ]

Sorry to bother you, but none of those conditions apply, we are not using a newer compiler and certainly not Linux:

Information for http://10.117.69.27/2015Q1/mongodb-3.0.3.tgz:
Build information:
ABI=64
BUILD_DATE=2015-07-10 21:36:25 +0000
BUILD_HOST=SunOS pkgsrc-pbulk-2014Q4-1.local 5.11 joyent_20141030T081701Z i86pc i386 i86pc
CATEGORIES=databases
CC_VERSION=gcc-4.9.2
CFLAGS=-O2 -pipe -O2 -I/opt/local/include -I/usr/include
CPPFLAGS= -fno-jump-tables -I/opt/local/include -I/usr/include
FFLAGS=-O
HOMEPAGE=http://mongodb.org/
LDFLAGS= -L/opt/local/gcc49/lib/gcc/x86_64-sun-solaris2.11/4.9.2 -Wl,-R/opt/local/gcc49/lib/gcc/x86_64-sun-solaris2.11/4.9.2 -L/opt/local/lib -Wl,-R/opt/local/lib -L/usr/lib/amd64 -Wl,-R/usr/lib/amd64 -lnsl -lsocket
LICENSE=gnu-agpl-v3
LOCALBASE=/opt/local
MACHINE_ARCH=x86_64
MACHINE_GNU_ARCH=x86_64
MAINTAINER=bartosz.kuzma@gmail.com
MONGODB_DBPATH=/var/mongodb
MONGODB_GROUP=mongodb
MONGODB_LOGPATH=/var/log/mongodb
MONGODB_USER=mongodb
NO_BIN_ON_CDROM=
NO_BIN_ON_FTP=
NO_SRC_ON_CDROM=
NO_SRC_ON_FTP=
OBJECT_FMT=ELF
OPSYS=SunOS
OS_VERSION=5.11
PKGGNUDIR=
PKGINFODIR=info
PKGMANDIR=man
PKGPATH=joyent/mongodb30
PKGTOOLS_VERSION=20091115
PKG_OPTIONS=ssl
PKG_SYSCONFBASEDIR=/opt/local/etc
PKG_SYSCONFDIR=/opt/local/etc
REQUIRES=/lib/amd64/libc.so.1
REQUIRES=/lib/amd64/libdl.so.1
REQUIRES=/lib/amd64/libm.so.2
REQUIRES=/lib/amd64/libnsl.so.1
REQUIRES=/lib/amd64/libpthread.so.1
REQUIRES=/lib/amd64/libresolv.so.2
REQUIRES=/lib/amd64/librt.so.1
REQUIRES=/lib/amd64/libsocket.so.1
REQUIRES=/lib/amd64/libumem.so.1
REQUIRES=/opt/local/gcc49/x86_64-sun-solaris2.11/lib/amd64/libgcc_s.so.1
REQUIRES=/opt/local/gcc49/x86_64-sun-solaris2.11/lib/amd64/libstdc++.so.6
REQUIRES=/opt/local/lib/libboost_filesystem.so.1.57.0
REQUIRES=/opt/local/lib/libboost_program_options.so.1.57.0
REQUIRES=/opt/local/lib/libboost_system.so.1.57.0
REQUIRES=/opt/local/lib/libboost_thread.so.1.57.0
REQUIRES=/opt/local/lib/libcrypto.so.1.0.0
REQUIRES=/opt/local/lib/libpcap.so.0
REQUIRES=/opt/local/lib/libpcre.so.1
REQUIRES=/opt/local/lib/libpcrecpp.so.0
REQUIRES=/opt/local/lib/libsnappy.so.1
REQUIRES=/opt/local/lib/libssl.so.1.0.0
REQUIRES=/usr/lib/amd64/liblgrp.so.1
RESTRICTED=
SSLBASE=/opt/local
SSLCERTS=/opt/local/etc/openssl/certs
SSLDIR=/opt/local/etc/openssl
SSLKEYS=/opt/local/etc/openssl/private
VARBASE=/var
_PLIST_IGNORE_FILES=
_USE_DESTDIR=user-destdir

Comment by Sam Kleinman (Inactive) [ 25/Aug/15 ]

Sorry for not getting back to you sooner.

We see errors around AdjustAmountOfExternalAllocatedMemory in situations with newer compilers (e.g. GCC 5.2) that are incompatible with the version of V8 embedded in MongoDB, as well as deployments with some SELinux/grsecurity policies affecting memory use. You could experiment with your security policy settings and and the standard build if you are not using this build already.

Starting in the 3.1.7 development release for the 3.2.0 stable release, MongoDB no longer embeds the V8 JavaScript engine, which should resolve this issue.

Also, you may have some luck addressing questions like these to the mongodb-user forum for help with additional troubleshooting and support. I'm going to go ahead and close this issue, but feel free to reopen if you have more information, or are able to reproduce without security policies that restrict memory use and the standard build.

Comment by Christopher Neill [ 25/Aug/15 ]

Requesting a status update.

Comment by Christopher Neill [ 25/Aug/15 ]

From Joyent Support:

Max Bruning (Joyent Support)
Aug 24, 16:46

Hi Chris,

From running, in the global zone:

$ dtrace -w -n 'proc:::signal-handle/pid == $target && args[0] == 11 && args[1]/

{print(*args[1]); ustack(); stop();}

' -p 70914
dtrace: description 'proc:::signal-handle' matched 1 probe
dtrace: allowing destructive actions
I see:

CPU ID FUNCTION:NAME
12 13728 psig:signal-handle siginfo_t {
int si_signo = 0xb
int si_code = 0x1
int si_errno = 0
int si_pad = 0
union __data = {
int [60] __pad = [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0x2000c003, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0xd1882e22, 0xddec3d, 0, 0, 0x520633fb, 0xdded95, 0x15d67519, 0x3, 0xa0350298, 0 ]
struct __proc = {
pid_t __pid = 0
union __pdata = {
struct __kill = {
uid_t __uid = 0
union sigval __value =

{ int sival_int = 0 void *sival_ptr = 0 }

}
struct __cld =

{ clock_t __utime = 0 int __status = 0 clock_t __stime = 0 }

}
ctid_t __ctid = 0
zoneid_t __zoneid = 0
}
struct __fault =

{ void *__addr = 0 int __trapno = 0 caddr_t __pc = 0 }

struct __file =

{ int __fd = 0 long __band = 0 }

struct __prof = {
caddr_t __faddr = 0
timestruc_t __tstamp =

{ time_t tv_sec = 0 long tv_nsec = 0 }

short __syscall = 0
char __nsysarg = '\0'
char __fault = '\0'
long [8] __sysarg = [ 0, 0x2000c003, 0, 0, 0, 0, 0, 0 ]
int [10] __mstate = [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ]
}
struct __rctl =

{ int32_t __entity = 0 }

}
}
mongod`_ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl+0x20
mongod`_ZN5boost6detail17sp_counted_impl_pIN5mongo10BSONHolderEE7disposeEv+0x34
mongod`_ZN5mongo10ObjTrackerINS_10BSONHolderEED1Ev+0x92
mongod`_ZN5mongo7V8ScopeD1Ev+0x161
mongod`_ZN5mongo7V8ScopeD0Ev+0x11
mongod`_ZN5boost6detail12shared_countD1Ev+0x39
mongod`_ZN5mongo11PooledScopeD1Ev+0xdc
mongod`_ZN5mongo11PooledScopeD0Ev+0x11
mongod`_ZN5mongo2mr5StateD1Ev+0x6f
mongod`_ZN5mongo2mr16MapReduceCommand3runEPNS_16OperationContextERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x14f9
mongod`_ZN5mongo12_execCommandEPNS_16OperationContextEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x2c
mongod`_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_iPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xecf
mongod`_ZN5mongo12_runCommandsEPNS_16OperationContextEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x57e
mongod`ZN5mongo8runQueryEPNS_16OperationContextERNS_7MessageERNS_12QueryMessageERKNS_15NamespaceStringERNS_5CurOpES3+0x1f45
mongod`_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xba4
mongod`_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xee
mongod`_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x321
libboost_thread.so.1.57.0`thread_proxy+0x66
libc.so.1`_thrp_setup+0x8a
libc.so.1`_lwp_start

After this, I ran

$ gcore 70914
Once gcore finished, I ran

$ prun 70914

to get mongod to continue from its stopped state.

Using mdb on the core file (and from the stack backtrace shown above via DTrace, it is dying at

mongod`_ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl+0x20
This is:

_ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl+0x20/ai
_ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl+0x20: cmpl $0x1,(%rax)

The %rax register should be a pointer to thread specific data, but it is null. mongod is dying because of using a null pointer.
The %rax register is used to pass return values from function calls.

_ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl+0x20::dis
_ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl: pushq %rbp
_ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl+1: movq %rsp,%rbp
_ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl+4: pushq %r13
_ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl+6: pushq %r12
_ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl+8: pushq %rbx
_ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl+9: movq %rdi,%r13
_ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl+0xc: subq $0x8,%rsp
_ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl+0x10:movq +0x62a7c1(%rip),%r12 <0x1b73798>
_ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl+0x17:movl (%r12),%edi
_ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl+0x1b:call +0x189ef0 <_ZN2v88internal6Thread14GetThreadLocalENS1_15LocalStorageKeyE>
_ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl+0x20:cmpl $0x1,(%rax)
The %rax value at this location is the return value from

call +0x189ef0 <_ZN2v88internal6Thread14GetThreadLocalENS1_15LocalStorageKeyE>
and _ZN2v88internal6Thread14GetThreadLocalENS1_15LocalStorageKeyE is:

_ZN2v88internal6Thread14GetThreadLocalENS1_15LocalStorageKeyE::dis
_ZN2v88internal6Thread14GetThreadLocalENS1_15LocalStorageKeyE: jmp -0xc5827d <PLT=libc.so.1`pthread_getspecific>

So pthread_getspecific is returning a NULL for some piece of thread specific data, and _ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl+0x20 tries to indirect through that causing a segv.

_ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl is:

> ::dem _ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl
_ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl == v8::V8::AdjustAmountOfExternalAllocatedMemory

Searching for "mongodb AdjustAmountOfExternalAllocatedMemory segmentation violation" shows a few hits, but not quite the same.

Comment by Christopher Neill [ 19/Aug/15 ]

FYI - per Joyent's recommendation, we doubled the size of the replica set members to go from 4GB to 8GB of RAM. Same thing:

[root@logger1 us-east-2 /var/log/mongodb]# grep crit mongodb-remote.log
2015-08-19T19:29:20+00:00 rs0db1 mongod.27018[33668]: [ID 702911 local7.crit] [conn399] Invalid access at address: 0#012
2015-08-19T19:29:20+00:00 rs0db1 mongod.27018[33668]: [ID 702911 local7.crit] [conn399] Got signal: 11 (Segmentation Fault).#012#012 0x14b5ef2 0x14b55a3 0x14b5cfe 0xfffffd7fff1679f6 0xfffffd7fff15a66b 0x1548fe0 0x1403454 0x14038d2 0x13fa4b1 0x13fa4e1 0xa807d9 0x13f334c 0x13f3de1 0xcae6af 0xcb2cb9 0xd4538c 0xd464df 0xd4733e 0xfca785 0xe83b54 0xa8555e 0x14571f1 0xfffffd7ff71b4816 0xfffffd7fff16761a 0xfffffd7fff167930#012----- BEGIN BACKTRACE -----#012{"backtrace":[

{"b":"400000","o":"10B5EF2"}

,

{"b":"400000","o":"10B55A3"}

,

{"b":"400000","o":"10B5CFE"}

,

{"b":"FFFFFD7FFF050000","o":"1179F6"}

,

{"b":"FFFFFD7FFF050000","o":"10A66B"}

,

{"b":"400000","o":"1148FE0"}

,

{"b":"400000","o":"1003454"}

,

{"b":"400000","o":"10038D2"}

,

{"b":"400000","o":"FFA4B1"}

,

{"b":"400000","o":"FFA4E1"}

,

{"b":"400000","o":"6807D9"}

,

{"b":"400000","o":"FF334C"}

,

{"b":"400000","o":"FF3DE1"}

,

{"b":"400000","o":"8AE6AF"}

,

{"b":"400000","o":"8B2CB9"}

,

{"b":"400000","o":"94538C"}

,

{"b":"400000","o":"9464DF"}

,

{"b":"400000","o":"94733E"}

,

{"b":"400000","o":"BCA785"}

,{"b":"400000","o"

Generated at Thu Feb 08 03:52:55 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.