[SERVER-16985] Segmentation fault in mongo, V8 Created: 21/Jan/15  Updated: 06/Mar/15  Resolved: 23/Jan/15

Status: Closed
Project: Core Server
Component/s: Shell, Stability
Affects Version/s: 2.6.7
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Darko Luketic Assignee: Andrew Morrow (Inactive)
Resolution: Duplicate Votes: 0
Labels: crash
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Gentoo Linux
Kernel 3.18.3 - gentoo-sources
GCC 4.9.2
glibc 2.20
boost 1.56.0
bundled V8


Attachments: File mongobug.tar.xz    
Issue Links:
Duplicate
duplicates SERVER-13824 V8-3.12 Segfaults when compiled with ... Closed
is duplicated by SERVER-17488 mongodb-2.6.[678] client startup fail... Closed
Operating System: Linux
Steps To Reproduce:

Run mongo on Gentoo Linux and try to connect to mongod

Participants:

 Description   

Related to SERVER-15920
Using the Gentoo ebuild.
On all 3 servers, Xeon E3-1245 & v2
with or without using or compiling with openssl
mongo crashes on connect in V8
Non-GRSEC enabled kernel
Additional information, without duplicating it here at
https://bugs.gentoo.org/show_bug.cgi?id=536760
trace is exactly the same

Please let me know if I need to add anything, missing libraries or anything else



 Comments   
Comment by Andrew Morrow (Inactive) [ 23/Jan/15 ]

Hi dluketic -

I believe you are encountering the issue described in SERVER-13824. This is a bug in our bundled v8 which fails to properly deal with type alignment. My guess is that the more aggressive instruction use enabled by -march=sandybridge or better is another path to triggering this particular issue.

We have patched the issue on master, but we have not backported it to the 2.6 branch. You can either:

  • Open a ticket requesting that we backport the fix for SERVER-13824 to 2.6.8. However, be aware that we are unlikely to make such a change this late in the lifecycle of 2.6.
  • Evaluate the patch in SERVER-13824 (https://github.com/mongodb/mongo/commit/0028a339876b352575758910a158973852ce4cfe) and integrate it with your Gentoo patch set. However, please note that while that patch does appear to solve the specific problem of the shell crashing on startup, we have reason to believe that there are other problems with v8 and GCC-4.9.
  • Avoid using optimization levels or codegen options (like -march=native) that result in the use of vectorized operations in v8.

Also, be aware that building against the system version of the libraries has other risks, as we cannot realistically validate the correctness of the database in all of these build configurations. In particular, we have some anecdotal evidence of issues building newer boost (1.57) with GCC-4.9.

Thanks,
Andrew

Comment by Darko Luketic [ 22/Jan/15 ]

The reason why it happens is
CFLAGS &| CXXFLAGS having
-march=native -mtune=native
or the equivalent of it, set because of distcc use.

-march=native
is enough to reproduce this bug
(or -march=sandybridge or -march=ivybridge)

distcc compilation also fails with an error (sorry no log available, was out of scope, just for the record)

Comment by Darko Luketic [ 22/Jan/15 ]

The 2 patches that are applied

### mongodb-2.6.2-fix-scons.patch ###
--- a/SConstruct        2014-06-16 19:32:46.274507258 +0200
+++ b/SConstruct        2014-06-16 19:34:04.763586829 +0200
@@ -813,7 +813,6 @@
     # -Winvalid-pch Warn if a precompiled header (see Precompiled Headers) is found in the search path but can't be used.
     env.Append( CCFLAGS=["-fPIC",
                          "-fno-strict-aliasing",
-                         "-ggdb",
                          "-pthread",
                          "-Wall",
                          "-Wsign-compare",
@@ -821,13 +820,13 @@
                          "-Winvalid-pch"] )
     # env.Append( " -Wconversion" ) TODO: this doesn't really work yet
     if linux or darwin:
-        env.Append( CCFLAGS=["-pipe"] )
         if not has_option("disable-warnings-as-errors"):
             env.Append( CCFLAGS=["-Werror"] )
 
     env.Append( CPPDEFINES=["_FILE_OFFSET_BITS=64"] )
-    env.Append( CXXFLAGS=["-Wnon-virtual-dtor", "-Woverloaded-virtual"] )
     env.Append( LINKFLAGS=["-fPIC", "-pthread"] )
+    env.Append( CXXFLAGS=os.environ['CXXFLAGS'] )
+    env.Append( LINKFLAGS=os.environ['LDFLAGS'] )
 
     # SERVER-9761: Ensure early detection of missing symbols in dependent libraries at program
     # startup.
@@ -844,7 +843,7 @@
     if not darwin:
         env.Append( LINKFLAGS=["-rdynamic"] )
 
-    env.Append( LIBS=[] )
+    env.Append( LIBS=['pcre', 'pcrecpp', 'snappy', 'yaml-cpp'] )
 
     #make scons colorgcc friendly
     for key in ('HOME', 'TERM'):
### mongodb-2.6.2-fix-scons.patch ###
 
### mongodb-2.4-fix-v8-pythonpath.patch ###
--- a/src/third_party/v8/SConscript     2014-05-26 11:00:37.725508098 +0200
+++ b/src/third_party/v8/SConscript     2014-05-26 11:01:13.265029083 +0200
@@ -29,7 +29,7 @@
 import sys
 from os.path import join, dirname, abspath
 root_dir = dirname(File('SConscript').rfile().abspath)
-sys.path.append(join(root_dir, 'tools'))
+sys.path.insert(0, join(root_dir, 'tools'))
 import js2c
 
 Import("env windows linux darwin solaris freebsd debugBuild")
### mongodb-2.4-fix-v8-pythonpath.patch ###

Comment by Darko Luketic [ 22/Jan/15 ]

pax-ctl -m
pax-ctl -r
pax-ctl -e
had no effect

Comment by Darko Luketic [ 22/Jan/15 ]

I have built just mongo without any options
scons mongo
and I'm able to connect to mongod without crashing

I will now try the various options
###
--usev8
succeeds, is able to connect without crashing

--use-system-boost --usev8 -j8
succeeds, is able to connect without crashing

--use-system-boost --use-system-pcre --usev8 -j8
succeeds, is able to connect without crashing

--use-system-boost --use-system-pcre --usev8 --use-system-snappy -j8
succeeds, is able to connect without crashing

--use-system-boost --use-system-pcre --usev8 --use-system-snappy --use-system-stemmer -j8
succeeds, is able to connect without crashing

--use-system-boost --use-system-pcre --usev8 --use-system-snappy --use-system-stemmer --use-system-tcmalloc -j8
succeeds, is able to connect without crashing

scons --use-system-boost --use-system-pcre --use-system-snappy --use-system-stemmer --use-system-tcmalloc --use-system-yaml --usev8 -j8 mongo
fails to build with

Target 'build/linux2/use-system-boost/use-system-pcre/use-system-snappy/use-system-stemmer/use-system-tcmalloc/use-system-yaml/usev8/mongo/mongo' depends on the availability of a system provided library for 'yaml', but no suitable library was found during configuration.
Target 'build/linux2/use-system-boost/use-system-pcre/use-system-snappy/use-system-stemmer/use-system-tcmalloc/use-system-yaml/usev8/mongo/mongo' depends on the availability of a system provided library for 'yaml', but no suitable library was found during configuration.
scons: *** [build/linux2/use-system-boost/use-system-pcre/use-system-snappy/use-system-stemmer/use-system-tcmalloc/use-system-yaml/usev8/mongo/mongo] Error 1
scons: building terminated because of errors.

after running
sed -i -e "s/[\"yaml\"]/[\"yaml-cpp\"]/" SConstruct || die

succeeds, is able to connect without crashing
###

so much for that

Generated at Thu Feb 08 03:42:55 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.