[SERVER-17447] Failed to build on Linux i686 with clang Created: 03/Mar/15  Updated: 07/Dec/16  Resolved: 27/Apr/15

Status: Closed
Project: Core Server
Component/s: Build
Affects Version/s: 3.0.0
Fix Version/s: 3.1.2

Type: Bug Priority: Major - P3
Reporter: Felix Yan Assignee: Unassigned
Resolution: Done Votes: 0
Labels: build-jbr
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File config.log    
Issue Links:
Related
related to SERVER-17455 Configure checks for __atomic and __s... Closed
related to SERVER-17461 32-bit clang cannot compile vendored ... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Steps To Reproduce:

The build command I was using:

scons all \
--use-system-pcre \
--use-system-snappy \
--use-system-tcmalloc \
--use-sasl-client \
--use-system-boost \
--variant-dir=build \
--ssl \
--disable-warnings-as-errors \
--cc=clang --cxx=clang++

Sprint: Platform 2 04/24/15, BUILD 2 04/24/15
Participants:

 Description   

I am getting the following errors when trying to build mongodb. A quick google shows it might need a i686 build target instead of i386?

clang++ -o build/build/mongo/db/auth/authorization_session.o -c -Wnon-virtual-dtor -Woverloaded-virtual -fPIC -fno-strict-aliasing -ggdb -pthread -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -pipe -O3 -march=nocona -mtune=generic -Wno-unused-function -Wno-unused-private-field -Wno-deprecated-declarations -Wno-tautological-constant-out-of-range-compare -Wno-unused-const-variable -Wno-missing-braces -fno-builtin-memcmp -D_SCONS -DMONGO_EXPOSE_MACROS -DSUPPORT_UTF8 -DMONGO_OPTIMIZED_BUILD -DMONGO_BYTE_ORDER=1234 -D_FILE_OFFSET_BITS=64 -DMONGO_SSL -DMONGO_HAVE___THREAD -DMONGO_HAVE_GCC_ATOMIC_BUILTINS -DBOOST_THREAD_VERSION=2 -DMONGO_HAVE_HEADER_UNISTD_H -DMONGO_HAVE_POSIX_MONOTONIC_CLOCK -DMONGO_HAVE_EXECINFO_BACKTRACE -Isrc/third_party/s2 -Ibuild/build -Isrc src/mongo/db/auth/authorization_session.cpp
build/build/mongo/client/dbclient.o: In function `mongo::AtomicIntrinsics<long long, void>::fetchAndAdd(long long volatile*, long long)':
/build/mongodb/src/mongodb-src-r3.0.0/src/mongo/platform/atomic_intrinsics_gcc_atomic.h:72: undefined reference to `__atomic_fetch_add_8'
build/build/mongo/client/dbclient_rs.o: In function `mongo::AtomicIntrinsics<long long, void>::fetchAndAdd(long long volatile*, long long)':
/build/mongodb/src/mongodb-src-r3.0.0/src/mongo/platform/atomic_intrinsics_gcc_atomic.h:72: undefined reference to `__atomic_fetch_add_8'
build/build/mongo/client/syncclusterconnection.o: In function `mongo::AtomicIntrinsics<long long, void>::fetchAndAdd(long long volatile*, long long)':
/build/mongodb/src/mongodb-src-r3.0.0/src/mongo/platform/atomic_intrinsics_gcc_atomic.h:72: undefined reference to `__atomic_fetch_add_8'
/build/mongodb/src/mongodb-src-r3.0.0/src/mongo/platform/atomic_intrinsics_gcc_atomic.h:72: undefined reference to `__atomic_fetch_add_8'
/build/mongodb/src/mongodb-src-r3.0.0/src/mongo/platform/atomic_intrinsics_gcc_atomic.h:72: undefined reference to `__atomic_fetch_add_8'
build/build/mongo/client/syncclusterconnection.o:/build/mongodb/src/mongodb-src-r3.0.0/src/mongo/platform/atomic_intrinsics_gcc_atomic.h:72: more undefined references to `__atomic_fetch_add_8' follow
clang++ -o build/build/mongo/db/auth/authz_documents_update_guard.o -c -Wnon-virtual-dtor -Woverloaded-virtual -fPIC -fno-strict-aliasing -ggdb -pthread -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -pipe -O3 -march=nocona -mtune=generic -Wno-unused-function -Wno-unused-private-field -Wno-deprecated-declarations -Wno-tautological-constant-out-of-range-compare -Wno-unused-const-variable -Wno-missing-braces -fno-builtin-memcmp -D_SCONS -DMONGO_EXPOSE_MACROS -DSUPPORT_UTF8 -DMONGO_OPTIMIZED_BUILD -DMONGO_BYTE_ORDER=1234 -D_FILE_OFFSET_BITS=64 -DMONGO_SSL -DMONGO_HAVE___THREAD -DMONGO_HAVE_GCC_ATOMIC_BUILTINS -DBOOST_THREAD_VERSION=2 -DMONGO_HAVE_HEADER_UNISTD_H -DMONGO_HAVE_POSIX_MONOTONIC_CLOCK -DMONGO_HAVE_EXECINFO_BACKTRACE -Isrc/third_party/s2 -Ibuild/build -Isrc src/mongo/db/auth/authz_documents_update_guard.cpp
clang: error: linker command failed with exit code 1 (use -v to see invocation)
scons: *** [build/build/mongo/mongo] Error 1
scons: building terminated because of errors.



 Comments   
Comment by Jonathan Reams [ 27/Apr/15 ]

Except for the upstream tcmalloc problem, this should be fixed now.

Comment by Andrew Morrow (Inactive) [ 23/Apr/15 ]

Hi felixonmars -

The LIBS=atomic workaround should no longer be needed on the master branch. Also, the tcmalloc issue has been filed upstream

Comment by Felix Yan [ 04/Mar/15 ]

Okay, then I'll use the workaround for now. Thanks!

Comment by Andrew Morrow (Inactive) [ 04/Mar/15 ]

Hi felixonmars -

32 bit support is not very high priority, but I expect that we will fix this in a reasonably timely manner. In the meantime, the LIBS=atomic workaround should be easily integrated into your packaging process, I expect.

Thanks,
Andrew

Comment by Andrew Morrow (Inactive) [ 04/Mar/15 ]

Also, I've filed SERVER-17461 to track the 32-bit clang errors in tcmalloc.

Comment by Felix Yan [ 04/Mar/15 ]

Ah, great. Thanks!

For this particular fix, I am not sure if this is the correct fix, but should I expect this to be fixed upstream?

Comment by Andrew Morrow (Inactive) [ 04/Mar/15 ]

You should just be able to say

scons LIBS=atomic ...

instead of needing to tunnel it in via the compiler.

Comment by Felix Yan [ 04/Mar/15 ]

Since I didn't find a sane way to specify the flag, i simply applied it to --cc and --cxx, and now it builds fine.

I mean:

--cc='clang -latomic' --cxx='clang++ -latomic'

Comment by Felix Yan [ 04/Mar/15 ]

I've played around your test program a bit, and looks like specifying -latomic could fix the linker error. Any thoughts?

e.g.:

$ clang++ --std=c++11 -latomic -m32 ./64_atomic.cpp

(finished without an error)

Comment by Andrew Morrow (Inactive) [ 04/Mar/15 ]

OK, that appears to confirm that we do not obtain 64-bit atomic operations when targeting 32-bit with clang, even in C++11 mode. I can confirm that on my system, with clang 3.5, the following fails to compile in 32-bit mode:

$ cat ./64_atomic.cpp
#include <atomic>
#include <cstdint>
 
int main(int argc, char* argv[]) {
    std::atomic<std::int64_t> x;
    return x.fetch_add(1);
}
 
$ clang++ --std=c++11 -m32 ./64_atomic.cpp
/tmp/64_atomic-31294e.o: In function `main':
./64_atomic.cpp:(.text+0x56): undefined reference to `__atomic_fetch_add_8'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
 
$ clang++ --version
Ubuntu clang version 3.5.0-4ubuntu2 (tags/RELEASE_350/final) (based on LLVM 3.5.0)
Target: x86_64-pc-linux-gnu
Thread model: posix

but the same program builds fine with GCC 4.8.2 in 32 bit mode. So, I think that if you want to target 32-bit, you will either need to get clang updated to a version offers the required intrinsics, or use GCC which as you point out is in conflict with SERVER-13880.

Comment by Felix Yan [ 04/Mar/15 ]

Hi acm

I have just tested with --allocator=system and it failed the same way with --system-tcmalloc as I have pasted above.

Comment by Andrew Morrow (Inactive) [ 04/Mar/15 ]

Hi felixonmars -

That is an interesting failure, but I think it is a distinct issue (clang, 32-bit, tcmalloc). I think we should try to sidestep that, for now, by disabling tcmalloc entirely with the --allocator=system flag, and see if we can get a definitive answer on clang std::atomic in 32-bit mode.

Thanks,
Andrew

Comment by Felix Yan [ 04/Mar/15 ]

This is the error when building with system tcmalloc and --c++11=on:

clang++ -o build/build/mongo/db/auth/authz_documents_update_guard.o -c -Wnon-virtual-dtor -Woverloaded-virtual -std=c++11 -fPIC -fno-strict-aliasing -ggdb -pthread -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -pipe -O3 -march=nocona -mtune=generic -Wno-unused-function -Wno-unused-private-field -Wno-deprecated-declarations -Wno-tautological-constant-out-of-range-compare -Wno-unused-const-variable -Wno-missing-braces -fno-builtin-memcmp -D_SCONS -DMONGO_EXPOSE_MACROS -DSUPPORT_UTF8 -DMONGO_OPTIMIZED_BUILD -DMONGO_BYTE_ORDER=1234 -D_FILE_OFFSET_BITS=64 -DMONGO_SSL -DMONGO_HAVE___THREAD -DMONGO_HAVE_CXX11_ATOMICS -DBOOST_THREAD_VERSION=2 -DMONGO_HAVE_HEADER_UNISTD_H -DMONGO_HAVE_POSIX_MONOTONIC_CLOCK -DMONGO_HAVE_EXECINFO_BACKTRACE -Isrc/third_party/s2 -Ibuild/build -Isrc src/mongo/db/auth/authz_documents_update_guard.cpp
build/build/mongo/client/dbclient.o: In function `std::__atomic_base<long long>::fetch_add(long long, std::memory_order)':
/bin/../lib/gcc/i686-pc-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/atomic_base.h:618: undefined reference to `__atomic_fetch_add_8'
build/build/mongo/client/dbclient_rs.o: In function `std::__atomic_base<long long>::fetch_add(long long, std::memory_order)':
/bin/../lib/gcc/i686-pc-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/atomic_base.h:618: undefined reference to `__atomic_fetch_add_8'
build/build/mongo/client/syncclusterconnection.o: In function `std::__atomic_base<long long>::fetch_add(long long, std::memory_order)':
/bin/../lib/gcc/i686-pc-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/atomic_base.h:618: undefined reference to `__atomic_fetch_add_8'
/bin/../lib/gcc/i686-pc-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/atomic_base.h:618: undefined reference to `__atomic_fetch_add_8'
/bin/../lib/gcc/i686-pc-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/atomic_base.h:618: undefined reference to `__atomic_fetch_add_8'
build/build/mongo/client/syncclusterconnection.o:/bin/../lib/gcc/i686-pc-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/atomic_base.h:618: more undefined references to `__atomic_fetch_add_8' follow
clang++ -o build/build/mongo/db/auth/authz_manager_external_state.o -c -Wnon-virtual-dtor -Woverloaded-virtual -std=c++11 -fPIC -fno-strict-aliasing -ggdb -pthread -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -pipe -O3 -march=nocona -mtune=generic -Wno-unused-function -Wno-unused-private-field -Wno-deprecated-declarations -Wno-tautological-constant-out-of-range-compare -Wno-unused-const-variable -Wno-missing-braces -fno-builtin-memcmp -D_SCONS -DMONGO_EXPOSE_MACROS -DSUPPORT_UTF8 -DMONGO_OPTIMIZED_BUILD -DMONGO_BYTE_ORDER=1234 -D_FILE_OFFSET_BITS=64 -DMONGO_SSL -DMONGO_HAVE___THREAD -DMONGO_HAVE_CXX11_ATOMICS -DBOOST_THREAD_VERSION=2 -DMONGO_HAVE_HEADER_UNISTD_H -DMONGO_HAVE_POSIX_MONOTONIC_CLOCK -DMONGO_HAVE_EXECINFO_BACKTRACE -Isrc/third_party/s2 -Ibuild/build -Isrc src/mongo/db/auth/authz_manager_external_state.cpp
clang: error: linker command failed with exit code 1 (use -v to see invocation)
scons: *** [build/build/mongo/mongo] Error 1
scons: building terminated because of errors.

Comment by Felix Yan [ 04/Mar/15 ]

--c++11=on failed with the following error:

clang++ -o build/build/third_party/gperftools-2.2/src/maybe_threads.o -c -Wnon-virtual-dtor -Woverloaded-virtual -std=c++11 -fPIC -fno-strict-aliasing -ggdb -pthread -Wno-unknown-pragmas -Winvalid-pch -pipe -O3 -march=nocona -mtune=generic -Wno-unused-function -Wno-unused-private-field -Wno-deprecated-declarations -Wno-tautological-constant-out-of-range-compare -Wno-unused-const-variable -Wno-missing-braces -fno-builtin-memcmp -D_SCONS -DMONGO_EXPOSE_MACROS -DSUPPORT_UTF8 -DMONGO_OPTIMIZED_BUILD -DMONGO_BYTE_ORDER=1234 -D_FILE_OFFSET_BITS=64 -DMONGO_SSL -DMONGO_HAVE___THREAD -DMONGO_HAVE_CXX11_ATOMICS -DBOOST_THREAD_VERSION=2 -DMONGO_HAVE_HEADER_UNISTD_H -DMONGO_HAVE_POSIX_MONOTONIC_CLOCK -DMONGO_HAVE_EXECINFO_BACKTRACE -DMALLOC_HOOK_MAYBE_VOLATILE=volatile -DNO_TCMALLOC_SAMPLES -DNO_HEAP_CHECK -DNDEBUG -Ibuild/build/third_party/gperftools-2.2/src -Isrc/third_party/gperftools-2.2/src -Isrc/third_party/gperftools-2.2 -Ibuild/build -Isrc src/third_party/gperftools-2.2/src/maybe_threads.cc
In file included from src/third_party/gperftools-2.2/src/malloc_hook.cc:687:
src/third_party/gperftools-2.2/src/malloc_hook_mmap_linux.h:108:23: error: non-constant-expression cannot be narrowed from type 'off_t' (aka 'long long') to 'int32' (aka 'int') in initializer list [-Wc++11-narrowing]
(off_t) offset };
^~~~~~~~~~~~~~
src/third_party/gperftools-2.2/src/malloc_hook_mmap_linux.h:108:23: note: insert an explicit cast to silence this issue
(off_t) offset };
^~~~~~~~~~~~~~
static_cast<int32>( )
1 error generated.
scons: *** [build/build/third_party/gperftools-2.2/src/malloc_hook.o] Error 1
scons: building terminated because of errors.

Comment by Felix Yan [ 04/Mar/15 ]

Update:

Sorry I think I misunderstood you - just built with GCC 4.9.2 and it did build successfully, so it should be clang missing 64-bit __atomic intrinsics.

I'll go on and test if --c++11=on helps.

Comment by Felix Yan [ 04/Mar/15 ]

I'm quickly answering some of them, and will test for others later:

  • Yes I am doing an intentional cross compile, as I am packaging the [community]/mongodb package for Arch, and Arch targets i686 (yes, it's 32-bit x86 architecture) and x86_64.. We are already providing a warning on installation time for the 32-bit limitations, too.
  • I am using clang because GCC is not an option: https://jira.mongodb.org/browse/SERVER-13880 (I note that the bug was marked as duplicate and the target bug has been fixed. But at least with latest 2.6.8 and GCC 4.9.2 I am still getting the segfault. I'll follow up that bug report later)
Comment by Andrew Morrow (Inactive) [ 04/Mar/15 ]

felixonmars

Thanks for the details. I have several comments and questions, and then a few suggestions:

  • First, I'd like to verify that what you are attempting here is an intentional cross compile. When you say i686 build, you mean the 32-bit x86 architecture, correct? If so, are you aware of the severe limitations on data capacity for 32-bit mongod builds? Additionally, the new WiredTiger storage engine is not offered in 32-bit builds. I'd go so far as to suggest that packaging a 32-bit mongo build may not really be worthwhile. A 32-bit build is only really useable for the most basic prototyping or experimenting. That said, this is intended to be a supported configuration, so...
  • Normally, I'd expect this combination of architecture, OS, toolchain, and standard library version to auto enable a C++11 build. However, you are building with system versions of several C++ libraries. The build system conservatively disables auto-enablement of C++11 mode if it sees that the build is configured to use system versions of C++ libraries (boost, etc.), since it cannot know if the system version of the library is built in C++11 mode, and some libraries, in particular boost, are not ABI stable between C++03 and C++11 mode compilation.
  • Since the build is not in C++11 mode, it does not attempt to use std::atomic, and instead falls back to the __atomic API. Note that in the configure output, there is an explicit check for support for at least the existence of the __atomic API, though not for all of the functions we use. This is intended as a coarse check, so that we can move on to looking for __sync if __atomic appears not to be offered.
  • Finally, note that the actual build error is a linker error: __atomic_fetch_add_8 is not found. This is a 64-bit atomic operation, and the fact that this symbol is being searched for at link time suggests that clang 3.5 may not offer this function as an intrinsic when targeting 32-bit x86 targets.

Given the above, you have a few different potential resolutions:

  • Stop packaging for 32-bit x86.
  • If you can confirm that your version of clang does not offer intrinsics for the 64-bit __atomic intrinsics you could try using GCC instead, if that is an option.
  • If arch is building the system version of its C++ libraries with --std=c++11, then you could try to explicitly build in C++11 mode by passing --c++11=on to scons so that std::atomic is used instead of the intrinsics. However, I wonder if the lack of an intrinsic for __atomic_fetch_add_8 on this target gives reason to suspect that std::atomic may also not work for 64-bit integers in this configuration.
  • If the system libraries on arch are not being built with C++11, you could instead use the vendored version of the third party libraries, which would also allow you to enable C++11, and potentially get you std::atomic, however, the same caveat exists there.

Also, it is probably an flaw in our build system that the check for __atomic does not actually invoke all of the __atomic functions that we require. If it had, then you would have had a failure at configure time, which presumably would have been more informative. I've opened SERVER-17455 to track this.

I realize this is a lot of detail and does not fully provide a path forward, but the situation is complex. Please let me know if you need clarification on any of the above points or suggestions.

For your question about documenting the status of --use-system-tcmalloc, I've filed DOCS-4907.

Thanks,
Andrew

Comment by Felix Yan [ 04/Mar/15 ]

Hi Andrea,

  • Host is latest Arch Linux x86_64, but I am building it in a clean Arch Linux i686 chroot (using Arch devtools)
  • Output of clang --version and clang++ --version:

    clang version 3.5.1 (tags/RELEASE_351/final)
    Target: i386-pc-linux-gnu
    Thread model: posix
    clang version 3.5.1 (tags/RELEASE_351/final)
    Target: i386-pc-linux-gnu
    Thread model: posix

  • Console output of "configure":

    scons: Reading SConscript files ...
    Mkdir("build/scons")
    scons version: 2.3.4
    python version: 2 7 9 'final' 0
    Checking whether the C++ compiler works... yes
    Checking whether the C compiler works... yes
    Checking if C++ compiler "clang++" is GCC... no
    Checking if C++ compiler "clang++" is clang... yes
    Checking if C compiler "clang" is clang... yes
    Checking if C compiler is clang 3.4 (or Apple XCode 5.1.1) or newer...yes
    Checking if C++ compiler is clang 3.4 (or Apple XCode 5.1.1) or newer...yes
    Checking if target architecture is 32-bit x86...yes
    WARNING: WiredTiger is not supported on 32-bit platforms, disabling support
    Checking if C compiler supports -Wno-unused-local-typedefs... no
    Checking if C compiler supports -Wno-unused-function... yes
    Checking if C compiler supports -Wno-unused-private-field... yes
    Checking if C compiler supports -Wno-deprecated-declarations... yes
    Checking if C compiler supports -Wno-tautological-constant-out-of-range-compare... yes
    Checking if C compiler supports -Wno-unused-const-variable... yes
    Checking if C compiler supports -Wno-unused-but-set-variable... no
    Checking if C compiler supports -Wno-missing-braces... yes
    Checking if implicit boost::shared_ptr null conversion is supported... yes
    Checking if we are using libstdc++... yes
    Checking for libstdc++ 4.6.0 or better... yes
    Checking if we are on a POSIX system... yes
    Checking if the POSIX monotonic clock is supported... yes
    Checking if C compiler supports -fno-builtin-memcmp... yes
    Checking for __thread... yes
    Checking for gcc __atomic builtins... yes
    Checking for C library pcre... yes
    Checking for C library pcrecpp... yes
    Checking for C library snappy... yes
    Checking for C++ header file boost/filesystem/operations.hpp... yes
    Checking for C++ library boost_thread-mt... no
    Checking for C++ library boost_thread... yes
    Checking for C++ library boost_filesystem-mt... no
    Checking for C++ library boost_filesystem... yes
    Checking for C++ library boost_program_options-mt... no
    Checking for C++ library boost_program_options... yes
    Checking for C++ library boost_system-mt... no
    Checking for C++ library boost_system... yes
    Checking for C library rt... yes
    Checking for C library dl... yes
    Checking for C++ header file execinfo.h... yes
    Checking whether backtrace is declared... yes
    Checking whether backtrace_symbols is declared... yes
    Checking whether backtrace_symbols_fd is declared... yes
    Checking for C library pcap... yes
    Checking for sasl_version_info(0, 0, 0, 0, 0, 0) in C library sasl2... yes
    Checking for C library tcmalloc... yes
    scons: done reading SConscript files.

  • config.log attached.

And thanks for the tips! Hope it will be documented in the Build section of the wiki, too (or if I have missed it?)

Comment by Andrew Morrow (Inactive) [ 03/Mar/15 ]

Hi felixonmars -

Could you please provide a few additional pieces of information to help us diagnose the issue?

  • The host architecture, operating system, and operating system version
  • The intended target architecture for the build, if different from the host architecture
  • The output of clang --version, and the output of clang++ --version
  • The console output of the "configure" step of a clean rebuild (should not say (cached) anywhere).
  • Please attach the file build/scons/config.log from that clean rebuild.

I'd also like to point out that building with --use-system-tcmalloc is not a recommended build configuration for MongoDB 3.0, as there are important customizations in our bundled version of tcmalloc.

Thanks,
Andrew

Generated at Thu Feb 08 03:44:28 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.