Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-13824

V8-3.12 Segfaults when compiled with GCC 4.9.0

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: JavaScript
    • Labels:
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Sprint:
      Server 2.7.3, Server 2.7.4

      Description

      2014-05-02T17:29:06.043-0400 mongo got signal 11 (Segmentation fault), stack trace:
      2014-05-02T17:29:06.048-0400 0x7d8e76 0x61fca9 0x7f40a6043df0 0xa44b9c 0xab1ff8 0xab2034 0xab2cd9 0xab6f1b 0xab73fa 0xa4b49e 0xa5b0ed 0xa50988 0xa5b7c6 0xa50a10 0xa5b7c6 0xa50a10 0xa5b7c6 0xa50a10 0xa5b7c6 0xa50a10
       /home/mstearn/10gen/mongo/mongo(_ZN5mongo15printStackTraceERSo+0x26) [0x7d8e76]
       /home/mstearn/10gen/mongo/mongo(_Z12quitAbruptlyi+0x109) [0x61fca9]
       /usr/lib/libc.so.6(+0x33df0) [0x7f40a6043df0]
       /home/mstearn/10gen/mongo/mongo(_ZN2v88internal6String11WriteToFlatItEEvPS1_PT_ii+0x3bc) [0xa44b9c]
       /home/mstearn/10gen/mongo/mongo(_ZN2v88internal33GenericStringUtf16CharacterStream10FillBufferEjj+0x38) [0xab1ff8]
       /home/mstearn/10gen/mongo/mongo(_ZN2v88internal28BufferedUtf16CharacterStream9ReadBlockEv+0x34) [0xab2034]
       /home/mstearn/10gen/mongo/mongo(_ZN2v88internal7Scanner20SkipMultiLineCommentEv+0x129) [0xab2cd9]
       /home/mstearn/10gen/mongo/mongo(_ZN2v88internal7Scanner4ScanEv+0x10eb) [0xab6f1b]
       /home/mstearn/10gen/mongo/mongo(_ZN2v88internal7Scanner4NextEv+0x4a) [0xab73fa]
       /home/mstearn/10gen/mongo/mongo(_ZN2v88internal6Parser6ExpectENS0_5Token5ValueEPb+0x5e) [0xa4b49e]
       /home/mstearn/10gen/mongo/mongo(_ZN2v88internal6Parser10ParseBlockEPNS0_8ZoneListINS0_6HandleINS0_6StringEEEEEPb+0x31d) [0xa5b0ed]
       /home/mstearn/10gen/mongo/mongo(_ZN2v88internal6Parser14ParseStatementEPNS0_8ZoneListINS0_6HandleINS0_6StringEEEEEPb+0x98) [0xa50988]
       /home/mstearn/10gen/mongo/mongo(_ZN2v88internal6Parser16ParseIfStatementEPNS0_8ZoneListINS0_6HandleINS0_6StringEEEEEPb+0x1d6) [0xa5b7c6]
       /home/mstearn/10gen/mongo/mongo(_ZN2v88internal6Parser14ParseStatementEPNS0_8ZoneListINS0_6HandleINS0_6StringEEEEEPb+0x120) [0xa50a10]
       /home/mstearn/10gen/mongo/mongo(_ZN2v88internal6Parser16ParseIfStatementEPNS0_8ZoneListINS0_6HandleINS0_6StringEEEEEPb+0x1d6) [0xa5b7c6]
       /home/mstearn/10gen/mongo/mongo(_ZN2v88internal6Parser14ParseStatementEPNS0_8ZoneListINS0_6HandleINS0_6StringEEEEEPb+0x120) [0xa50a10]
       /home/mstearn/10gen/mongo/mongo(_ZN2v88internal6Parser16ParseIfStatementEPNS0_8ZoneListINS0_6HandleINS0_6StringEEEEEPb+0x1d6) [0xa5b7c6]
       /home/mstearn/10gen/mongo/mongo(_ZN2v88internal6Parser14ParseStatementEPNS0_8ZoneListINS0_6HandleINS0_6StringEEEEEPb+0x120) [0xa50a10]
       /home/mstearn/10gen/mongo/mongo(_ZN2v88internal6Parser16ParseIfStatementEPNS0_8ZoneListINS0_6HandleINS0_6StringEEEEEPb+0x1d6) [0xa5b7c6]
       /home/mstearn/10gen/mongo/mongo(_ZN2v88internal6Parser14ParseStatementEPNS0_8ZoneListINS0_6HandleINS0_6StringEEEEEPb+0x120) [0xa50a10]

        Issue Links

          Activity

          Hide
          matt.kangas Matt Kangas (Inactive) added a comment -

          Demangled stack trace:

           /home/mstearn/10gen/mongo/mongo(mongo::printStackTrace(std::basic_ostream<char, std::char_traits<char> >&)+0x26) [0x7d8e76]
           /home/mstearn/10gen/mongo/mongo(quitAbruptly(int)+0x109) [0x61fca9]
           /usr/lib/libc.so.6(+0x33df0) [0x7f40a6043df0]
           /home/mstearn/10gen/mongo/mongo(void v8::internal::String::WriteToFlat<unsigned short>(v8::internal::String*, unsigned short*, int, int)+0x3bc) [0xa44b9c]
           /home/mstearn/10gen/mongo/mongo(v8::internal::GenericStringUtf16CharacterStream::FillBuffer(unsigned int, unsigned int)+0x38) [0xab1ff8]
           /home/mstearn/10gen/mongo/mongo(v8::internal::BufferedUtf16CharacterStream::ReadBlock()+0x34) [0xab2034]
           /home/mstearn/10gen/mongo/mongo(v8::internal::Scanner::SkipMultiLineComment()+0x129) [0xab2cd9]
           /home/mstearn/10gen/mongo/mongo(v8::internal::Scanner::Scan()+0x10eb) [0xab6f1b]
           /home/mstearn/10gen/mongo/mongo(v8::internal::Scanner::Next()+0x4a) [0xab73fa]
           /home/mstearn/10gen/mongo/mongo(v8::internal::Parser::Expect(v8::internal::Token::Value, bool*)+0x5e) [0xa4b49e]
           /home/mstearn/10gen/mongo/mongo(v8::internal::Parser::ParseBlock(v8::internal::ZoneList<v8::internal::Handle<v8::internal::String> >*, bool*)+0x31d) [0xa5b0ed]
           /home/mstearn/10gen/mongo/mongo(v8::internal::Parser::ParseStatement(v8::internal::ZoneList<v8::internal::Handle<v8::internal::String> >*, bool*)+0x98) [0xa50988]
           /home/mstearn/10gen/mongo/mongo(v8::internal::Parser::ParseIfStatement(v8::internal::ZoneList<v8::internal::Handle<v8::internal::String> >*, bool*)+0x1d6) [0xa5b7c6]
           /home/mstearn/10gen/mongo/mongo(v8::internal::Parser::ParseStatement(v8::internal::ZoneList<v8::internal::Handle<v8::internal::String> >*, bool*)+0x120) [0xa50a10]
           /home/mstearn/10gen/mongo/mongo(v8::internal::Parser::ParseIfStatement(v8::internal::ZoneList<v8::internal::Handle<v8::internal::String> >*, bool*)+0x1d6) [0xa5b7c6]
           /home/mstearn/10gen/mongo/mongo(v8::internal::Parser::ParseStatement(v8::internal::ZoneList<v8::internal::Handle<v8::internal::String> >*, bool*)+0x120) [0xa50a10]
           /home/mstearn/10gen/mongo/mongo(v8::internal::Parser::ParseIfStatement(v8::internal::ZoneList<v8::internal::Handle<v8::internal::String> >*, bool*)+0x1d6) [0xa5b7c6]
           /home/mstearn/10gen/mongo/mongo(v8::internal::Parser::ParseStatement(v8::internal::ZoneList<v8::internal::Handle<v8::internal::String> >*, bool*)+0x120) [0xa50a10]
           /home/mstearn/10gen/mongo/mongo(v8::internal::Parser::ParseIfStatement(v8::internal::ZoneList<v8::internal::Handle<v8::internal::String> >*, bool*)+0x1d6) [0xa5b7c6]
           /home/mstearn/10gen/mongo/mongo(v8::internal::Parser::ParseStatement(v8::internal::ZoneList<v8::internal::Handle<v8::internal::String> >*, bool*)+0x120) [0xa50a10]

          Show
          matt.kangas Matt Kangas (Inactive) added a comment - Demangled stack trace: /home/mstearn/10gen/mongo/mongo(mongo::printStackTrace(std::basic_ostream<char, std::char_traits<char> >&)+0x26) [0x7d8e76] /home/mstearn/10gen/mongo/mongo(quitAbruptly(int)+0x109) [0x61fca9] /usr/lib/libc.so.6(+0x33df0) [0x7f40a6043df0] /home/mstearn/10gen/mongo/mongo(void v8::internal::String::WriteToFlat<unsigned short>(v8::internal::String*, unsigned short*, int, int)+0x3bc) [0xa44b9c] /home/mstearn/10gen/mongo/mongo(v8::internal::GenericStringUtf16CharacterStream::FillBuffer(unsigned int, unsigned int)+0x38) [0xab1ff8] /home/mstearn/10gen/mongo/mongo(v8::internal::BufferedUtf16CharacterStream::ReadBlock()+0x34) [0xab2034] /home/mstearn/10gen/mongo/mongo(v8::internal::Scanner::SkipMultiLineComment()+0x129) [0xab2cd9] /home/mstearn/10gen/mongo/mongo(v8::internal::Scanner::Scan()+0x10eb) [0xab6f1b] /home/mstearn/10gen/mongo/mongo(v8::internal::Scanner::Next()+0x4a) [0xab73fa] /home/mstearn/10gen/mongo/mongo(v8::internal::Parser::Expect(v8::internal::Token::Value, bool*)+0x5e) [0xa4b49e] /home/mstearn/10gen/mongo/mongo(v8::internal::Parser::ParseBlock(v8::internal::ZoneList<v8::internal::Handle<v8::internal::String> >*, bool*)+0x31d) [0xa5b0ed] /home/mstearn/10gen/mongo/mongo(v8::internal::Parser::ParseStatement(v8::internal::ZoneList<v8::internal::Handle<v8::internal::String> >*, bool*)+0x98) [0xa50988] /home/mstearn/10gen/mongo/mongo(v8::internal::Parser::ParseIfStatement(v8::internal::ZoneList<v8::internal::Handle<v8::internal::String> >*, bool*)+0x1d6) [0xa5b7c6] /home/mstearn/10gen/mongo/mongo(v8::internal::Parser::ParseStatement(v8::internal::ZoneList<v8::internal::Handle<v8::internal::String> >*, bool*)+0x120) [0xa50a10] /home/mstearn/10gen/mongo/mongo(v8::internal::Parser::ParseIfStatement(v8::internal::ZoneList<v8::internal::Handle<v8::internal::String> >*, bool*)+0x1d6) [0xa5b7c6] /home/mstearn/10gen/mongo/mongo(v8::internal::Parser::ParseStatement(v8::internal::ZoneList<v8::internal::Handle<v8::internal::String> >*, bool*)+0x120) [0xa50a10] /home/mstearn/10gen/mongo/mongo(v8::internal::Parser::ParseIfStatement(v8::internal::ZoneList<v8::internal::Handle<v8::internal::String> >*, bool*)+0x1d6) [0xa5b7c6] /home/mstearn/10gen/mongo/mongo(v8::internal::Parser::ParseStatement(v8::internal::ZoneList<v8::internal::Handle<v8::internal::String> >*, bool*)+0x120) [0xa50a10] /home/mstearn/10gen/mongo/mongo(v8::internal::Parser::ParseIfStatement(v8::internal::ZoneList<v8::internal::Handle<v8::internal::String> >*, bool*)+0x1d6) [0xa5b7c6] /home/mstearn/10gen/mongo/mongo(v8::internal::Parser::ParseStatement(v8::internal::ZoneList<v8::internal::Handle<v8::internal::String> >*, bool*)+0x120) [0xa50a10]
          Hide
          matt.kangas Matt Kangas (Inactive) added a comment -

          In SERVER-13880 a user reported that ArchLinux's build of 2.6.1-1 (using gcc 4.9) produced a similar failure:

          Just type "db.abc.ensureIndex(
          {'m':1}
          )" in mongo-shell.

          We need do a full test suite run against v2.6 built with gcc-4.9 and see what falls out.

          Show
          matt.kangas Matt Kangas (Inactive) added a comment - In SERVER-13880 a user reported that ArchLinux's build of 2.6.1-1 (using gcc 4.9) produced a similar failure: Just type "db.abc.ensureIndex( {'m':1} )" in mongo-shell. We need do a full test suite run against v2.6 built with gcc-4.9 and see what falls out.
          Hide
          adam.midvidy Adam Midvidy (Inactive) added a comment - - edited

          Further investigation shows that this is probably due to an unaligned access occurring in vectorized code,

          Here is the code generated with -O2 (no crash occurs)

                *reinterpret_cast<uintptr_t*>(dest) =
            a3c398:	48 8b 0a             	mov    (%rdx),%rcx
                    *reinterpret_cast<const uintptr_t*>(src);
                dest += kStepSize;
            a3c39b:	48 83 c0 08          	add    $0x8,%rax
                src += kStepSize;
            a3c39f:	48 83 c2 08          	add    $0x8,%rdx
                return;
              }
              // Number of characters in a uintptr_t.
              static const int kStepSize = sizeof(uintptr_t) / sizeof(*dest);  // NOLINT
              while (dest <= limit - kStepSize) { }}

          Here is the same code compiled with -O3 (the crash occurs at instruction 0xa66b7c)

             // Number of characters in a uintptr_t.
              static const int kStepSize = sizeof(uintptr_t) / sizeof(*dest);  // NOLINT
              while (dest <= limit - kStepSize) {
                *reinterpret_cast<uintptr_t*>(dest) =
            a66b7c:	66 41 0f 6f 44 15 00 	movdqa 0x0(%r13,%rdx,1),%xmm0
            a66b83:	49 83 c0 01          	add    $0x1,%r8
            a66b87:	41 0f 11 04 12       	movups %xmm0,(%r10,%rdx,1)
            a66b8c:	48 83 c2 10          	add    $0x10,%rdx
            a66b90:	4d 39 c8             	cmp    %r9,%r8
          }

          Note that the loop has been vectorized to use the xmm registers using the movdqa instruction. From the intel arch documentation: "When the source or destination operand is a memory operand, the operand must be aligned on a 16-byte boundary or a general-protection exception (#GP) will be generated." As there is no requirement that the input to CopyChars be aligned, the use of the xmm registers triggers the fault.

          Furthermore, clang with -O3 does not emit vector instructions for the function in question.

          Show
          adam.midvidy Adam Midvidy (Inactive) added a comment - - edited Further investigation shows that this is probably due to an unaligned access occurring in vectorized code, Here is the code generated with -O2 (no crash occurs) *reinterpret_cast<uintptr_t*>(dest) = a3c398: 48 8b 0a mov (%rdx),%rcx *reinterpret_cast<const uintptr_t*>(src); dest += kStepSize; a3c39b: 48 83 c0 08 add $0x8,%rax src += kStepSize; a3c39f: 48 83 c2 08 add $0x8,%rdx return; } // Number of characters in a uintptr_t. static const int kStepSize = sizeof(uintptr_t) / sizeof(*dest); // NOLINT while (dest <= limit - kStepSize) { }} Here is the same code compiled with -O3 (the crash occurs at instruction 0xa66b7c) // Number of characters in a uintptr_t. static const int kStepSize = sizeof(uintptr_t) / sizeof(*dest); // NOLINT while (dest <= limit - kStepSize) { *reinterpret_cast<uintptr_t*>(dest) = a66b7c: 66 41 0f 6f 44 15 00 movdqa 0x0(%r13,%rdx,1),%xmm0 a66b83: 49 83 c0 01 add $0x1,%r8 a66b87: 41 0f 11 04 12 movups %xmm0,(%r10,%rdx,1) a66b8c: 48 83 c2 10 add $0x10,%rdx a66b90: 4d 39 c8 cmp %r9,%r8 } Note that the loop has been vectorized to use the xmm registers using the movdqa instruction. From the intel arch documentation: "When the source or destination operand is a memory operand, the operand must be aligned on a 16-byte boundary or a general-protection exception (#GP) will be generated." As there is no requirement that the input to CopyChars be aligned, the use of the xmm registers triggers the fault. Furthermore, clang with -O3 does not emit vector instructions for the function in question.
          Hide
          matt.kangas Matt Kangas (Inactive) added a comment -

          GCC 4.9.1 has been released and appears to resolve this issue. We are no longer able to reproduce the segfault.

          If you must compile with GCC 4.9.0, recommended workarounds include:

          • Compile with -O2 rather than -O3
          • Compile with -fno-tree-vectorize

          Note: our SConstruct does not provide a way to specify alternate compiler options, so you have to hack this line to specify "-O2" rather than "-O3".
          https://github.com/mongodb/mongo/blame/14ec31c/SConstruct#L908

          Show
          matt.kangas Matt Kangas (Inactive) added a comment - GCC 4.9.1 has been released and appears to resolve this issue. We are no longer able to reproduce the segfault. If you must compile with GCC 4.9.0, recommended workarounds include: Compile with -O2 rather than -O3 Compile with -fno-tree-vectorize Note: our SConstruct does not provide a way to specify alternate compiler options, so you have to hack this line to specify "-O2" rather than "-O3". https://github.com/mongodb/mongo/blame/14ec31c/SConstruct#L908
          Hide
          geert.bosch Geert Bosch added a comment - - edited

          I tried gcc (GCC) 4.9.2 20140725 (prerelease) and it failed in similar fashion. As GCC developer I wanted to extract a bug report for GCC (if it was indeed a bug in GCC) or otherwise find out what was wrong. A look at the sources in v8utils.h clearly indicates the problem. The assumption is that it is OK to use reinterpret_cast to convert between pointer types. This is undefined in general, but entirely unreasonable if the types have different alignment as is the case here. Indeed the whole purpose of the code is to ignore alignment. True, you might get away with it in some cases, but that doesn't make it correct.

          Note that the whole notion of "V8_HOST_CAN_READ_UNALIGNED" is flawed. Can read what unaligned? An integer? A floating point double? A 32-byte vector? Answers can differ, and the V8 code is utterly broken in this regard. Unfortunately, even if V8_HOST_CAN_READ_UNALIGNED is false, there is a reinterpret_cast of pointers to uint32_t (even on 64-bit systems!) to try and manually optimize loops for which GCC/glibc would use optimal code if memcmp had simply been called instead of CompareRawStringContents.

          I have a patch that includes a minimal fix to v8 that essentially omits setting V8_HOST_CAN_READ_UNALIGNED and fixes CompareRawStringContents by removing the code that tries to compare by words.

          -Geert

          Show
          geert.bosch Geert Bosch added a comment - - edited I tried gcc (GCC) 4.9.2 20140725 (prerelease) and it failed in similar fashion. As GCC developer I wanted to extract a bug report for GCC (if it was indeed a bug in GCC) or otherwise find out what was wrong. A look at the sources in v8utils.h clearly indicates the problem. The assumption is that it is OK to use reinterpret_cast to convert between pointer types. This is undefined in general, but entirely unreasonable if the types have different alignment as is the case here. Indeed the whole purpose of the code is to ignore alignment. True, you might get away with it in some cases, but that doesn't make it correct. Note that the whole notion of "V8_HOST_CAN_READ_UNALIGNED" is flawed. Can read what unaligned? An integer? A floating point double? A 32-byte vector? Answers can differ, and the V8 code is utterly broken in this regard. Unfortunately, even if V8_HOST_CAN_READ_UNALIGNED is false, there is a reinterpret_cast of pointers to uint32_t (even on 64-bit systems!) to try and manually optimize loops for which GCC/glibc would use optimal code if memcmp had simply been called instead of CompareRawStringContents. I have a patch that includes a minimal fix to v8 that essentially omits setting V8_HOST_CAN_READ_UNALIGNED and fixes CompareRawStringContents by removing the code that tries to compare by words. -Geert
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'name': u'Geert Bosch', u'email': u'geert@mongodb.com'}

          Message: SERVER-13824: Remove some typing violations from the v8 code

          Code will now work correctly with GCC 4.9.x with full optimization.
          Branch: master
          https://github.com/mongodb/mongo/commit/0028a339876b352575758910a158973852ce4cfe

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'name': u'Geert Bosch', u'email': u'geert@mongodb.com'} Message: SERVER-13824 : Remove some typing violations from the v8 code Code will now work correctly with GCC 4.9.x with full optimization. Branch: master https://github.com/mongodb/mongo/commit/0028a339876b352575758910a158973852ce4cfe

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                  Agile