Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-91716

SCons invocations with VARIANT_DIR/DESTDIR/NINJA_PREFIX impact each other

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Unresolved
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Build
    • ALL

      For my day-to-day work I need to build two flavors of the server - -dbg=on/-opt=off for better debugging and a dist_test one for running tests locally. For the purpose I have always generated two different ninja.build files. Up until recently these have been completely isolated from each other (this bug doesn't reproduce in 8.0), but starting recently they seem to impact each other and in a very subtle way.

      Starting from a clean repository, running the following will build the debug flavor of the server:

      buildscripts/scons.py --ssl --variables-files=etc/scons/developer_versions.vars --variables-files=etc/scons/mongodbtoolchain_stable_clang.vars -j128 --jlink=4 ICECC=icecc CCACHE=ccache VARIANT_DIR=ninja DESTDIR=build/install_ninja --link-model=dynamic --dbg=on --opt=off --ninja generate-ninja

ninja -j128 install-all

      If now, without changing any source files I generate a different ninja file (note the different VARIANT_DIR/DESTDIR/NINJA_PREFIX-es for the SCons invocation and that the ninja invocation is the same as above), this causes the ninja build from above to rebuild everything:

      buildscripts/scons.py --ssl --variables-files=etc/scons/developer_versions.vars --variables-files=etc/scons/mongodbtoolchain_stable_gcc.vars -j128 --jlink=4 ICECC=icecc CCACHE=ccache --link-model=dynamic VARIANT_DIR=dist_test DESTDIR=build/install_dist_test --separate-debug=on GDB_INDEX=0 --ninja NINJA_PREFIX=dist_test generate-ninja

ninja -j128 install-all
...
[0/6652] Running bazel build
INFO: Invocation ID: ca3ef5fd-3373-41e2-abd0-08209df55c40
INFO: Streaming build results to: https://sodalite.cluster.engflow.com/invocation/ca3ef5fd-3373-41e2-abd0-08209df55c40
INFO: Analyzed 575 targets (0 packages loaded, 0 targets configured).
INFO: Found 575 targets...
INFO: Writing explanation of rebuilds to 'bazel_explain.log'
INFO: Elapsed time: 48.332s, Critical Path: 42.58s
INFO: 2218 processes: 1680 remote cache hit, 538 internal.
INFO: Streaming build results to: https://sodalite.cluster.engflow.com/invocation/ca3ef5fd-3373-41e2-abd0-08209df55c40
INFO: Build completed successfully, 2218 total actions
[67/6652] Compiled build/ninja/mongo/crypto/jwk_manager.dyn.o

      What is worse is, that if I let that second ninja build to run, it later on fails with this linking error:

      [3985/6587] Linked build/ninja/mongo/db/modules/enterprise/src/streams/third_party/mongocxx/libbsoncxx.so                             (2 results) [1581/1929]
FAILED: build/ninja/mongo/db/modules/enterprise/src/streams/third_party/mongocxx/libbsoncxx.so
rm -f build/ninja/mongo/db/modules/enterprise/src/streams/third_party/mongocxx/libbsoncxx.so; export CCACHE_NOCPP2='1';export CCACHE_PREFIX='/home/ubuntu/wor
kspace/mongo/build/scons/icecream/ninja/run-icecc.sh';export ICECC_CLANG_REMOTE_CPP='1';export PATH='/opt/mongodbtoolchain/v4/bin:/usr/local/bin:/opt/bin:/bi
n:/usr/bin';/bin/icerun /opt/mongodbtoolchain/v4/bin/clang++ @build/ninja/mongo/db/modules/enterprise/src/streams/third_party/mongocxx/libbsoncxx.so.rsp && t
ouch build/ninja/mongo/db/modules/enterprise/src/streams/third_party/mongocxx/libbsoncxx.so.gdb-index && /opt/mongodbtoolchain/v4/bin/gdb --batch-silent --qu
iet --nx --eval-command "save gdb-index build/ninja/mongo/db/modules/enterprise/src/streams/third_party/mongocxx" build/ninja/mongo/db/modules/enterprise/src
/streams/third_party/mongocxx/libbsoncxx.so && /opt/mongodbtoolchain/v4/bin/objcopy --add-section .gdb_index=build/ninja/mongo/db/modules/enterprise/src/streams/third_party/mongocxx/libbsoncxx.so.gdb-index --set-section-flags .gdb_index=readonly build/ninja/mongo/db/modules/enterprise/src/streams/third_party/mong
ocxx/libbsoncxx.so build/ninja/mongo/db/modules/enterprise/src/streams/third_party/mongocxx/libbsoncxx.so && rm -f build/ninja/mongo/db/modules/enterprise/sr
c/streams/third_party/mongocxx/libbsoncxx.so.gdb-index
ld.lld: error: undefined symbol: boost::assertion_failed(char const*, char const*, char const*, long)
>>> referenced by optional.hpp:1270 (src/third_party/boost/boost/optional/optional.hpp:1270)
>>>               build/ninja/mongo/db/modules/enterprise/src/streams/third_party/mongocxx/dist/bsoncxx/string/view_or_value.dyn.o:(boost::optional<std::__cx
x11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::get())
clang-12: error: linker command failed with exit code 1 (use -v to see invocation)
[3991/6587] Linked build/ninja/mongo/libbase.so                                                                                                              FAILED: build/ninja/mongo/libbase.so
rm -f build/ninja/mongo/libbase.so; export CCACHE_NOCPP2='1';export CCACHE_PREFIX='/home/ubuntu/workspace/mongo/build/scons/icecream/ninja/run-icecc.sh';export ICECC_CLANG_REMOTE_CPP='1';export PATH='/opt/mongodbtoolchain/v4/bin:/usr/local/bin:/opt/bin:/bin:/usr/bin';/bin/icerun /opt/mongodbtoolchain/v4/bin/clang
++ @build/ninja/mongo/libbase.so.rsp && touch build/ninja/mongo/libbase.so.gdb-index && /opt/mongodbtoolchain/v4/bin/gdb --batch-silent --quiet --nx --eval-c
ommand "save gdb-index build/ninja/mongo" build/ninja/mongo/libbase.so && /opt/mongodbtoolchain/v4/bin/objcopy --add-section .gdb_index=build/ninja/mongo/lib
base.so.gdb-index --set-section-flags .gdb_index=readonly build/ninja/mongo/libbase.so build/ninja/mongo/libbase.so && rm -f build/ninja/mongo/libbase.so.gdb
-index
ld.lld: error: undefined symbol: boost::assertion_failed(char const*, char const*, char const*, long)
>>> referenced by intrusive_ptr.hpp:199 (src/third_party/boost/boost/smart_ptr/intrusive_ptr.hpp:199)
>>>               build/ninja/mongo/base/error_codes.dyn.o:(boost::intrusive_ptr<mongo::Status::ErrorInfo const>::operator->() const)
>>> referenced by intrusive_ptr.hpp:199 (src/third_party/boost/boost/smart_ptr/intrusive_ptr.hpp:199)
>>>               build/ninja/mongo/base/initializer.dyn.o:(boost::intrusive_ptr<mongo::allocator_aware::SharedBuffer<std::allocator<void> >::Holder>::operat
or->() const)
>>> referenced by optional.hpp:1270 (src/third_party/boost/boost/optional/optional.hpp:1270)
>>>               build/ninja/mongo/base/parse_number.dyn.o:(boost::optional<unsigned long>::get())
>>> referenced 95 more times

ld.lld: error: undefined symbol: mongo::BoostAssertFuncs::global()
>>> referenced by boost_assert_impl.cpp:41 (src/mongo/util/boost_assert_impl.cpp:41)
>>>               build/ninja/mongo/util/boost_assert_impl.dyn.o:(mongo::BoostAssertImpl::BoostAssertImpl())
>>> referenced by boost_assert_impl.cpp:46 (src/mongo/util/boost_assert_impl.cpp:46)
>>>               build/ninja/mongo/util/boost_assert_impl.dyn.o:(mongo::BoostAssertImpl::BoostAssertImpl())

      This seems to have been broken within 2-3 days from the filing of this ticket and doesn't reproduce on 8.0.

            Assignee:
            Unassigned Unassigned
            Reporter:
            kaloian.manassiev@mongodb.com Kaloian Manassiev
            Votes:
            2 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated: