[SERVER-12991] Segmentation fault during V8 initialization on grsecurity Linux kernel Created: 03/Mar/14  Updated: 11/Aug/15  Resolved: 28/Mar/14

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 2.6.0-rc0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Miroslav Zacek Assignee: Jonathan Reams
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File mongo_crash.tgz    
Issue Links:
Duplicate
is duplicated by SERVER-15920 Segmentation fault on mongod processe... Closed
is duplicated by SERVER-19248 Segmentation fault running query with... Closed
Related
related to SERVER-16564 package installation should configure... Closed
Operating System: ALL
Steps To Reproduce:

1. Use kernel GR patched kernel: http://molly.corsac.net/~corsac/debian/kernel-grsec/packages/wheezy/
add

deb http://molly.corsac.net/~corsac/debian/kernel-grsec/packages/ wheezy/

run

apt-get install linux-image-3.2.45.1-grsec

and reboot

2. Install mongodb-org from http://downloads-distro.mongodb.org:
add

deb http://downloads-distro.mongodb.org/repo/debian-sysvinit dist 10gen

to apt sources
and run

apt-get install mongodb-org

3. run

mongo

Participants:

 Description   

On GRSEC + PAX patched kernel the mongo client crashes (segmentation fault)

2014-03-03T11:11:46.662+0100 WARNING: Cannot detect if NUMA interleaving is enabled. Failed to probe "/sys/devices/system/node/node1": Permission denied
MongoDB shell version: 2.6.0-rc0
mongo got signal 11 (Segmentation fault), stack trace: 
0x85cd81 0x61a182 0x6e3c65a034f0 0xb3fdb5 0xc0eabf 0xb40dde 0xb40df9 0xbdc82e 0xb217c2 0xbdc960 0x952ad7 0xbe337a 0x7a0175 0x7b2959 0x61d9d9 0x6203a3 0x6e3c659efead 0x618d49 
 mongo(_ZN5mongo15printStackTraceERSo+0x21) [0x85cd81]
 mongo(_Z12quitAbruptlyi+0x3b2) [0x61a182]
 /lib/x86_64-linux-gnu/libc.so.6(+0x324f0) [0x6e3c65a034f0]
 mongo(_ZN2v88internal2OS8AllocateEmPmb+0xe5) [0xb3fdb5]
 mongo(_ZN2v88internal28CreateTranscendentalFunctionENS0_19TranscendentalCache4TypeE+0x3f) [0xc0eabf]
 mongo(_ZN2v88internal22init_fast_sin_functionEv+0xe) [0xb40dde]
 mongo(_ZN2v88internal14POSIXPostSetUpEv+0x9) [0xb40df9]
 mongo(_ZN2v88internal2V828InitializeOncePerProcessImplEv+0x3e) [0xbdc82e]
 mongo(_ZN2v88internal12CallOnceImplEPlPFvPvES2_+0x62) [0xb217c2]
 mongo(_ZN2v88internal2V810InitializeEPNS0_12DeserializerE+0x20) [0xbdc960]
 mongo(_ZN2v82V810InitializeEv+0x47) [0x952ad7]
 mongo(_ZN2v86LockerC1EPNS_7IsolateE+0x9a) [0xbe337a]
 mongo(_ZN5mongo7V8ScopeC1EPNS_14V8ScriptEngineE+0x405) [0x7a0175]
 mongo(_ZN5mongo14V8ScriptEngine11createScopeEv+0x29) [0x7b2959]
 mongo(_Z5_mainiPPcS0_+0x449) [0x61d9d9]
 mongo(main+0x33) [0x6203a3]
 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd) [0x6e3c659efead]
 mongo(__gxx_personality_v0+0x459) [0x618d49]

I've experienced the same error with the 2.5.5 version. I've installed the unstable version because previous versions failed on accessing /sys/devices which is banned by GRSEC patch. Server is now able to start but the client crashes.

system: Debian wheezy
kernel: 3.2.45.1-grsec #1 SMP Mon May 27 13:14:33 CEST 2013 x86_64 GNU/Linux
memory: 1GB
installed packages:

ii  mongodb-org                              2.6.0~rc0                     amd64        MongoDB cross-platform document-oriented database system (metapackage)
ii  mongodb-org-mongos                       2.6.0~rc0                     amd64        MongoDB sharding router
ii  mongodb-org-server                       2.6.0~rc0                     amd64        MongoDB database server
ii  mongodb-org-shell                        2.6.0~rc0                     amd64        MongoDB shell client
ii  mongodb-org-tools                        2.6.0~rc0                     amd64        MongoDB tools

The tail of the strace log around the crash:

open("/proc/version", O_RDONLY)         = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x722a27d62000
read(3, "Linux version 3.2.45.1-grsec (ro"..., 1024) = 124
close(3)                                = 0
munmap(0x722a27d62000, 4096)            = 0
open("/proc/meminfo", O_RDONLY)         = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x722a27d62000
read(3, "MemTotal:        1020104 kB\nMemF"..., 1024) = 1024
close(3)                                = 0
munmap(0x722a27d62000, 4096)            = 0
getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=4*1024}) = 0
write(1, "MongoDB shell version: 2.6.0-rc0"..., 33MongoDB shell version: 2.6.0-rc0
) = 33
open("/dev/urandom", O_RDONLY)          = 3
read(3, "0\265\34\1\310\27\346N\2\352!\251$\246\302\35\252\23\203{\10\375\21\376\201\333o5/\366\264e"..., 8191) = 8191
close(3)                                = 0
mmap(NULL, 8392704, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x722a26399000
mprotect(0x722a26399000, 4096, PROT_NONE) = 0
clone(child_stack=0x722a26b98fd0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x722a26b999d0, tls=0x722a26b99700, child_tidptr=0x722a26b999d0) = 6208
getrlimit(RLIMIT_DATA, {rlim_cur=RLIM_INFINITY, rlim_max=RLIM_INFINITY}) = 0
gettimeofday({1393844278, 60308}, NULL) = 0
mmap(0x238c04db1000, 4096, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x722a26398000
mmap(0x722a26398000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = -1 EPERM (Operation not permitted)
munmap(0x722a26398000, 4096)            = 0
mmap(0x32cca9915000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 EPERM (Operation not permitted)
--- SIGSEGV (Segmentation fault) @ 0 (0) ---



 Comments   
Comment by Matt Kangas [ 28/Mar/14 ]

V8, the Javascript engine used by mongodb, requires the ability to write executable pages of memory for Just-In-Time (JIT) compilation.

If an operating system has been configured so it's not possible to write to executable memory regions, V8 cannot function.

The impact to MongoDB in this case is:

  • mongo shell: fails immediately on invocation
  • mongod: only functional if the scripting engine is disabled (--noscripting)
  • mongos: not affected

It is possible that mongod may start successfully but will fail as soon as the scripting engine is used.

For Linux grsecurity patched kernels, the recommended workaround is to set the following PaX flags.

	MPROTECT is disabled
	RANDEXEC is disabled
	EMUTRAMP is disabled

Comment by Jonathan Reams [ 28/Mar/14 ]

Also see the application-specific settings for grsecurity:
http://en.wikibooks.org/wiki/Grsecurity/Application-specific_Settings#Google_Chrome_15.0.874.106

On the section for google chrome - which also uses V8, you have to turn off the mprotect patches for the chrome to get it to work properly.

Comment by Jonathan Reams [ 23/Mar/14 ]

As an experiment, I patched the LOG macro so that logging would be initialized if it wasn't already - thus getting past this particular failure. That just moved the segfault down the road a bit:

Starting program: /home/vagrant/mongo-r2.6.0-rc2/build/linux2/d/mongo/mongo 
2014-03-23T19:26:37.176+0000 WARNING: Cannot detect if NUMA interleaving is enabled. Failed to probe "/sys/devices/system/node/node1": Permission denied
MongoDB shell version: 2.6.0-rc2
 
Program received signal SIGSEGV, Segmentation fault.
0x0000000000cdf400 in v8::internal::FixedArray::get (this=0x0, index=45) at src/third_party/v8/src/objects-inl.h:1683
1683	  return READ_FIELD(this, kHeaderSize + index * kPointerSize);
(gdb) bt
#0  0x0000000000cdf400 in v8::internal::FixedArray::get (this=0x0, index=45) at src/third_party/v8/src/objects-inl.h:1683
#1  0x0000000000deb07c in v8::internal::Heap::ClearJSFunctionResultCaches (this=0x416a010) at src/third_party/v8/src/heap.cc:745
#2  0x0000000000de9c15 in v8::internal::Heap::GarbageCollectionPrologue (this=0x416a010) at src/third_party/v8/src/heap.cc:395
#3  0x0000000000deaa4f in v8::internal::Heap::CollectGarbage (this=0x416a010, space=v8::internal::CODE_SPACE, collector=v8::internal::MARK_COMPACTOR, 
    gc_reason=0x11ff6b9 "allocation failure", collector_reason=0x1205230 "GC in old space requested") at src/third_party/v8/src/heap.cc:575
#4  0x0000000000d4c69e in v8::internal::Heap::CollectGarbage (this=0x416a010, space=v8::internal::CODE_SPACE, gc_reason=0x11ff6b9 "allocation failure")
    at src/third_party/v8/src/heap-inl.h:440
#5  0x0000000000db5eb8 in v8::internal::Factory::NewCode (this=0x416a000, desc=..., flags=-16128, self_ref=..., immovable=false) at src/third_party/v8/src/factory.cc:868
#6  0x0000000000d46870 in v8::internal::CodeStub::GetCode (this=0x72e5ad7392e0) at src/third_party/v8/src/code-stubs.cc:118
#7  0x0000000000def8b9 in v8::internal::Heap::CreateJSEntryStub (this=0x416a010) at src/third_party/v8/src/heap.cc:2538
#8  0x0000000000def9e3 in v8::internal::Heap::CreateFixedStubs (this=0x416a010) at src/third_party/v8/src/heap.cc:2562
#9  0x0000000000df0356 in v8::internal::Heap::CreateInitialObjects (this=0x416a010) at src/third_party/v8/src/heap.cc:2717
#10 0x0000000000df8f8a in v8::internal::Heap::SetUp (this=0x416a010, create_heap_objects=true) at src/third_party/v8/src/heap.cc:6147
#11 0x0000000000e8f8ed in v8::internal::Isolate::Init (this=0x416a000, des=0x0) at src/third_party/v8/src/isolate.cc:1863
#12 0x0000000001018d3e in v8::internal::V8::Initialize (des=0x0) at src/third_party/v8/src/v8.cc:93
#13 0x0000000000ce2921 in v8::InitializeHelper () at src/third_party/v8/src/api.cc:296
#14 0x0000000000cfe2cb in v8::V8::Initialize () at src/third_party/v8/src/api.cc:4243
#15 0x000000000101f1a3 in v8::Locker::Locker (this=0x72e5ad7396b0, isolate=0x416a000) at src/third_party/v8/src/v8threads.cc:66
#16 0x0000000000af0e5a in mongo::V8Scope::V8Scope (this=0x40b8000, engine=0x4180100) at src/mongo/scripting/engine_v8.cpp:481
#17 0x0000000000afd82f in mongo::V8ScriptEngine::createScope (this=0x4180100) at src/mongo/scripting/engine_v8.h:485
#18 0x0000000000aea86f in mongo::ScriptEngine::newScope (this=0x4180100) at src/mongo/scripting/engine.h:182
#19 0x00000000009bf996 in _main (argc=1, argv=0x72e5ad73a148, envp=0x72e5ad73a158) at src/mongo/shell/dbshell.cpp:744
#20 0x00000000009c146e in main (argc=1, argv=0x72e5ad73a148, envp=0x72e5ad73a158) at src/mongo/shell/dbshell.cpp:980

Comment by Jonathan Reams [ 23/Mar/14 ]

This was easily reproducible with the provided vagrant box. The segfault happens because logging isn't initialized when LOG is called after the allocation failure. I was able to get mongo to start relatively successfully by setting the correct PaX flags that disable some of the grsec protections:

vagrant@dev:~/mongo-r2.6.0-rc2$ /sbin/paxctl -cm build/linux2/dd/mongo/mongo
file build/linux2/dd/mongo/mongo had a PT_GNU_STACK program header, converted
vagrant@dev:~/mongo-r2.6.0-rc2$ /sbin/paxctl -v build/linux2/dd/mongo/mongo
PaX control v0.7
Copyright 2004,2005,2006,2007,2009,2010,2011,2012 PaX Team <pageexec@freemail.hu>
 
- PaX flags: -----m-x-e-- [build/linux2/dd/mongo/mongo]
	MPROTECT is disabled
	RANDEXEC is disabled
	EMUTRAMP is disabled
vagrant@dev:~/mongo-r2.6.0-rc2$ build/linux2/dd/mongo/mongo
2014-03-23T18:59:17.231+0000 WARNING: Cannot detect if NUMA interleaving is enabled. Failed to probe "/sys/devices/system/node/node1": Permission denied
MongoDB shell version: 2.6.0-rc2
connecting to: test
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
	http://docs.mongodb.org/
Questions? Try the support group
	http://groups.google.com/group/mongodb-user
Server has startup warnings: 
2014-03-23T17:44:07.297+0000 [initandlisten] 
2014-03-23T17:44:07.297+0000 [initandlisten] ** WARNING: Cannot detect if NUMA interleaving is enabled. Failed to probe "/sys/devices/system/node/node1": Permission denied
2014-03-23T17:44:07.298+0000 [initandlisten] unable to validate readahead settings due to error: boost::filesystem::status: Permission denied: "/sys/dev/block/254:0/queue/read_ahead_kb"
2014-03-23T17:44:07.298+0000 [initandlisten] for more information, see http://dochub.mongodb.org/core/readahead
> exit
bye

Comment by Miroslav Zacek [ 17/Mar/14 ]
  1. unpack the attached mongo_crash.tgz

    tar -xzf mongo_crash.tgz

  2. start up the virtual machine

    cd mongo_crash
    vagrant up

  3. reboot the virtual machine (to use the secured kernel)

    vagrant reload --no-provision

  4. run mongo

    vagrant ssh
    mongo

Comment by Matt Kangas [ 10/Mar/14 ]

According to the addr2line output, the segfault occurred during V8 initialization. Specifically:

The mmap() does not succeed on your system. While attempting to log the message "mmap failed", a segfault occurs. I suspect it's because i::Isolate::Current() returns NULL.

If it wasn't for the LOG failure, the rest of this code seems like it should handle the mmap() failure with fallback transcendental functions.

In any event, the steps to resolving this are:

  1. We will first have to reproduce the failure locally
  2. Then search upstream V8 history for a fix we can backport
  3. Failing that, roll our own fix

Can you point us to a Vagrant configuration (or something similar) which makes it easy to reproduce this failure?

Comment by Matt Kangas [ 04/Mar/14 ]

addr2line output from release artifacts with debug symbols.

$ addr2line -pifCe mongo 0x85cd81 0x61a182 0x6e3c65a034f0 0xb3fdb5 0xc0eabf 0xb40dde 0xb40df9 0xbdc82e 0xb217c2 0xbdc960 0x952ad7 0xbe337a 0x7a0175 0x7b2959 0x61d9d9 0x6203a3 0x6e3c659efead 0x618d49
mongo::printStackTrace(std::ostream&) at /srv/10gen/mci-exec/mci/git@github.commongodb/mongo.git/master/src/mongo/util/stacktrace.cpp:304
std::basic_streambuf<char, std::char_traits<char> >::pptr() const at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/streambuf:515
 (inlined by) std::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::str() const at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/sstream:131
 (inlined by) std::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::str() const at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/sstream:463
 (inlined by) quitAbruptly(int) at /srv/10gen/mci-exec/mci/git@github.commongodb/mongo.git/master/src/mongo/shell/dbshell.cpp:234
??
??:0
v8::internal::OS::Allocate(unsigned long, unsigned long*, bool) at /srv/10gen/mci-exec/mci/git@github.commongodb/mongo.git/master/src/third_party/v8/src/platform-linux.cc:342
v8::internal::CreateTranscendentalFunction(v8::internal::TranscendentalCache::Type) at /srv/10gen/mci-exec/mci/git@github.commongodb/mongo.git/master/src/third_party/v8/src/x64/codegen-x64.cc:64
v8::internal::init_fast_sin_function() at /srv/10gen/mci-exec/mci/git@github.commongodb/mongo.git/master/src/third_party/v8/src/platform-posix.cc:141
v8::internal::POSIXPostSetUp() at /srv/10gen/mci-exec/mci/git@github.commongodb/mongo.git/master/src/third_party/v8/src/platform-posix.cc:332
v8::internal::V8::InitializeOncePerProcessImpl() at /srv/10gen/mci-exec/mci/git@github.commongodb/mongo.git/master/src/third_party/v8/src/v8.cc:271
Release_Store at /srv/10gen/mci-exec/mci/git@github.commongodb/mongo.git/master/src/third_party/v8/src/atomicops_internals_x86_gcc.h:230
 (inlined by) v8::internal::CallOnceImpl(long*, void (*)(void*), void*) at /srv/10gen/mci-exec/mci/git@github.commongodb/mongo.git/master/src/third_party/v8/src/once.cc:62
v8::internal::Isolate::CurrentPerIsolateThreadData() at /srv/10gen/mci-exec/mci/git@github.commongodb/mongo.git/master/src/third_party/v8/src/isolate.h:435
 (inlined by) v8::internal::V8::Initialize(v8::internal::Deserializer*) at /srv/10gen/mci-exec/mci/git@github.commongodb/mongo.git/master/src/third_party/v8/src/v8.cc:73
InitializeHelper at /srv/10gen/mci-exec/mci/git@github.commongodb/mongo.git/master/src/third_party/v8/src/api.cc:296
 (inlined by) v8::V8::Initialize() at /srv/10gen/mci-exec/mci/git@github.commongodb/mongo.git/master/src/third_party/v8/src/api.cc:4243
Locker at /srv/10gen/mci-exec/mci/git@github.commongodb/mongo.git/master/src/third_party/v8/src/v8threads.cc:67
V8Scope at /srv/10gen/mci-exec/mci/git@github.commongodb/mongo.git/master/src/mongo/scripting/engine_v8.cpp:482
mongo::V8ScriptEngine::createScope() at /srv/10gen/mci-exec/mci/git@github.commongodb/mongo.git/master/src/mongo/scripting/engine_v8.h:485
_main(int, char**, char**) at /srv/10gen/mci-exec/mci/git@github.commongodb/mongo.git/master/src/mongo/shell/dbshell.cpp:736
main at /srv/10gen/mci-exec/mci/git@github.commongodb/mongo.git/master/src/mongo/shell/dbshell.cpp:969
??
??:0
_start at ??:0

Comment by Matt Kangas [ 03/Mar/14 ]

Hmm, is it really gperftools? That's the only other reference to /sys/devices that we haven't explicitly guarded.

$ git grep '\/devices'
src/mongo/db/startup_warnings.cpp:            hasMultipleNumaNodes = boost::filesystem::exists("/sys/devices/system/node/node1");
src/mongo/util/processinfo_linux2.cpp:            hasMultipleNodes = boost::filesystem::exists("/sys/devices/system/node/node1");
src/third_party/gperftools-2.0/src/base/sysinfo.cc:      ReadIntFromFile("/sys/devices/system/cpu/cpu0/tsc_freq_khz", &freq)) {
src/third_party/gperftools-2.0/src/base/sysinfo.cc:      ReadIntFromFile("/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq",

Code here:
https://github.com/mongodb/mongo/blame/master/src/third_party/gperftools-2.0/src/base/sysinfo.cc#L298-L310

Comment by Matt Kangas [ 03/Mar/14 ]

Note, this did not occur on a stock Linux kernel for any distro, but a specific security-patched kernel (grsecurity)

Related: SERVER-12480, SERVER-10623, SERVER-12464 which occurred with OVH's custom kernel.

Comment by Matt Kangas [ 03/Mar/14 ]

Demangled stack trace:

 mongo(mongo::printStackTrace(std::basic_ostream<char, std::char_traits<char> >&)+0x21) [0x85cd81]
 mongo(quitAbruptly(int)+0x3b2) [0x61a182]
 /lib/x86_64-linux-gnu/libc.so.6(+0x324f0) [0x6e3c65a034f0]
 mongo(v8::internal::OS::Allocate(unsigned long, unsigned long*, bool)+0xe5) [0xb3fdb5]
 mongo(v8::internal::CreateTranscendentalFunction(v8::internal::TranscendentalCache::Type)+0x3f) [0xc0eabf]
 mongo(v8::internal::init_fast_sin_function()+0xe) [0xb40dde]
 mongo(v8::internal::POSIXPostSetUp()+0x9) [0xb40df9]
 mongo(v8::internal::V8::InitializeOncePerProcessImpl()+0x3e) [0xbdc82e]
 mongo(v8::internal::CallOnceImpl(long*, void (*)(void*), void*)+0x62) [0xb217c2]
 mongo(v8::internal::V8::Initialize(v8::internal::Deserializer*)+0x20) [0xbdc960]
 mongo(v8::V8::Initialize()+0x47) [0x952ad7]
 mongo(v8::Locker::Locker(v8::Isolate*)+0x9a) [0xbe337a]
 mongo(mongo::V8Scope::V8Scope(mongo::V8ScriptEngine*)+0x405) [0x7a0175]
 mongo(mongo::V8ScriptEngine::createScope()+0x29) [0x7b2959]
 mongo(_main(int, char**, char**)+0x449) [0x61d9d9]
 mongo(main+0x33) [0x6203a3]

Generated at Thu Feb 08 03:30:18 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.