[SERVER-45848] Generated ninja file can try to link some objects before it's dependencies are ready Created: 29/Jan/20  Updated: 02/Feb/20  Resolved: 31/Jan/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Mathew Robinson (Inactive) Assignee: Mathew Robinson (Inactive)
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-45479 Missing ninja build generator for tes... Closed
Operating System: ALL
Sprint: Dev Platform 2020-02-24
Participants:

 Description   

[4644/4886] Building build/debug/mongo/db/matcher/libexpressions.so
FAILED: build/debug/mongo/db/matcher/libexpressions.so 
PATH=/opt/mongodbtoolchain/v3/bin:/usr/local/bin:/opt/bin:/bin:/usr/bin CCACHE_NOCPP2=1 ICECC_VERSION=/data/build/scons/icecc/3bd3951bba3f1291547efbcc21098663.tar.gz CCACHE_PREFIX=/bin/icecc /bin/icerun /opt/mongodbtoolchain/v3/bin/g++ @/home/chasinglogic/Work/mongo/build/debug/response_files/tmpbx3uaji3.lnk
/opt/mongodbtoolchain/revisions/91aa820a4e15ec424e214cdc861c1f4aa575a72c/stow/gcc-v3.fox/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../x86_64-mongodb-linux/bin/ld.gold: error: cannot open build/debug/third_party/pcre-8.42/libpcrecpp.so: No such file or directory
/opt/mongodbtoolchain/revisions/91aa820a4e15ec424e214cdc861c1f4aa575a72c/stow/gcc-v3.fox/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../x86_64-mongodb-linux/bin/ld.gold: error: cannot open build/debug/third_party/boost-1.70.0/libboost_filesystem.so: No such file or directory
src/mongo/db/matcher/expression_leaf.cpp:252: error: undefined reference to 'pcrecpp::RE::no_arg'
src/mongo/db/matcher/expression_leaf.cpp:252: error: undefined reference to 'pcrecpp::RE::no_arg'
src/mongo/db/matcher/expression_leaf.cpp:252: error: undefined reference to 'pcrecpp::RE::no_arg'
src/mongo/db/matcher/expression_leaf.cpp:252: error: undefined reference to 'pcrecpp::RE::no_arg'
src/mongo/db/matcher/expression_leaf.cpp:252: error: undefined reference to 'pcrecpp::RE::PartialMatch(pcrecpp::StringPiece const&, pcrecpp::Arg const&, pcrecpp::Arg const&, pcrecpp::Arg const&, pcrecpp::Arg const&, pcrecpp::Arg const&, pcrecpp::Arg const&, pcrecpp::Arg const&, pcrecpp::Arg const&, pcrecpp::Arg const&, pcrecpp::Arg const&, pcrecpp::Arg const&, pcrecpp::Arg const&, pcrecpp::Arg const&, pcrecpp::Arg const&, pcrecpp::Arg const&, pcrecpp::Arg const&) const'
src/third_party/pcre-8.42/pcrecpp.h:494: error: undefined reference to 'pcrecpp::RE::Init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, pcrecpp::RE_Options const*)'
/opt/mongodbtoolchain/revisions/91aa820a4e15ec424e214cdc861c1f4aa575a72c/stow/gcc-v3.fox/include/c++/8.2.0/bits/unique_ptr.h:81: error: undefined reference to 'pcrecpp::RE::~RE()'
src/third_party/pcre-8.42/pcrecpp.h:491: error: undefined reference to 'pcrecpp::RE::Init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, pcrecpp::RE_Options const*)'
src/mongo/db/matcher/schema/expression_internal_schema_allowed_properties.cpp:109: error: undefined reference to 'pcrecpp::RE::PartialMatch(pcrecpp::StringPiece const&, pcrecpp::Arg const&, pcrecpp::Arg const&, pcrecpp::Arg const&, pcrecpp::Arg const&, pcrecpp::Arg const&, pcrecpp::Arg const&, pcrecpp::Arg const&, pcrecpp::Arg const&, pcrecpp::Arg const&, pcrecpp::Arg const&, pcrecpp::Arg const&, pcrecpp::Arg const&, pcrecpp::Arg const&, pcrecpp::Arg const&, pcrecpp::Arg const&, pcrecpp::Arg const&) const'
collect2: error: ld returned 1 exit status

Occurs when running with high j values and icecream building the

{all}

target.



 Comments   
Comment by Mathew Robinson (Inactive) [ 31/Jan/20 ]

This was caused by SCons being used by ninja to make the test lists. We recently made the test lists depend on the tests so somewhere deep in the tree ninja would ask scons to build the test list which would cause SCons to try to re-build all the tests and their dependencies while ninja was trying to compile other files which required those same dependencies.

SCons removes files before building hence the "file not found errors" and many of the invalid ELF and stranger errors were probably caused by trying to link mid-compile by SCons.

Now that we've taught Ninja how to make these files it no longer invokes SCons to do so.

Comment by Mathew Robinson (Inactive) [ 29/Jan/20 ]

For reference this is the failing build definition from above:

build build/debug/mongo/db/matcher/libexpressions.so: CMD $
     build/debug/mongo/db/matcher/expression.dyn.o $
     build/debug/mongo/db/matcher/expression_algo.dyn.o $
     build/debug/mongo/db/matcher/expression_array.dyn.o $
     build/debug/mongo/db/matcher/expression_expr.dyn.o $
     build/debug/mongo/db/matcher/expression_geo.dyn.o $
     build/debug/mongo/db/matcher/expression_internal_expr_eq.dyn.o $
     build/debug/mongo/db/matcher/expression_leaf.dyn.o $
     build/debug/mongo/db/matcher/expression_parser.dyn.o $
     build/debug/mongo/db/matcher/expression_text_base.dyn.o $
     build/debug/mongo/db/matcher/expression_text_noop.dyn.o $
     build/debug/mongo/db/matcher/expression_tree.dyn.o $
     build/debug/mongo/db/matcher/expression_where_base.dyn.o $
     build/debug/mongo/db/matcher/expression_where_noop.dyn.o $
     build/debug/mongo/db/matcher/expression_with_placeholder.dyn.o $
     build/debug/mongo/db/matcher/extensions_callback.dyn.o $
     build/debug/mongo/db/matcher/extensions_callback_noop.dyn.o $
     build/debug/mongo/db/matcher/match_details.dyn.o build/debug/mongo/db/matcher/matchable.dyn.o $
     build/debug/mongo/db/matcher/matcher.dyn.o $
     build/debug/mongo/db/matcher/matcher_type_set.dyn.o $
     build/debug/mongo/db/matcher/rewrite_expr.dyn.o $
     build/debug/mongo/db/matcher/schema/encrypt_schema_types.dyn.o $
     build/debug/mongo/db/matcher/schema/expression_internal_schema_all_elem_match_from_index.dyn.o $
     build/debug/mongo/db/matcher/schema/expression_internal_schema_allowed_properties.dyn.o $
     build/debug/mongo/db/matcher/schema/expression_internal_schema_cond.dyn.o $
     build/debug/mongo/db/matcher/schema/expression_internal_schema_eq.dyn.o $
     build/debug/mongo/db/matcher/schema/expression_internal_schema_fmod.dyn.o $
     build/debug/mongo/db/matcher/schema/expression_internal_schema_match_array_index.dyn.o $
     build/debug/mongo/db/matcher/schema/expression_internal_schema_num_array_items.dyn.o $
     build/debug/mongo/db/matcher/schema/expression_internal_schema_num_properties.dyn.o $
     build/debug/mongo/db/matcher/schema/expression_internal_schema_object_match.dyn.o $
     build/debug/mongo/db/matcher/schema/expression_internal_schema_root_doc_eq.dyn.o $
     build/debug/mongo/db/matcher/schema/expression_internal_schema_str_length.dyn.o $
     build/debug/mongo/db/matcher/schema/expression_internal_schema_unique_items.dyn.o $
     build/debug/mongo/db/matcher/schema/expression_internal_schema_xor.dyn.o $
     build/debug/mongo/db/matcher/schema/json_pointer.dyn.o $
     build/debug/mongo/db/matcher/schema/json_schema_parser.dyn.o $
     build/debug/mongo/db/matcher/schema/encrypt_schema_gen.dyn.o | $
     build/debug/mongo/db/matcher/schema/expression_internal_schema_eq.dyn.o $
     build/debug/mongo/scripting/libscripting.so build/debug/mongo/db/query/libquery_knobs.so $
     build/debug/mongo/db/matcher/schema/expression_internal_schema_str_length.dyn.o $
     build/debug/mongo/db/storage/libstorage_options.so $
     build/debug/mongo/db/matcher/extensions_callback.dyn.o $
     build/debug/mongo/db/geo/libgeoparser.so build/debug/mongo/util/libfail_point.so $
     build/debug/mongo/db/matcher/expression_leaf.dyn.o $
     build/debug/mongo/db/fts/libfts_query_noop.so build/debug/third_party/pcre-8.42/libpcrecpp.so $
     build/debug/mongo/db/libnamespace_string.so build/debug/mongo/rpc/libclient_metadata.so $
     build/debug/mongo/util/libsecure_compare_memory.so $
     build/debug/mongo/db/liblogical_time_validator.so $
53     build/debug/mongo/db/libtime_proof_service.so $
      build/debug/third_party/timelib-2018.01/libtimelib.so $
      build/debug/mongo/db/matcher/schema/expression_internal_schema_match_array_index.dyn.o $
      build/debug/mongo/db/matcher/expression_algo.dyn.o build/debug/mongo/client/libsasl_client.so $
      build/debug/third_party/libshim_allocator.so build/debug/mongo/client/libclient_query.so $
      build/debug/mongo/rpc/librpc.so build/debug/mongo/db/storage/libwrite_unit_of_work.so $
      build/debug/mongo/db/matcher/expression_text_base.dyn.o $
      build/debug/mongo/util/libintrusive_counter.so $
      build/debug/mongo/db/repl/libreplication_process.so $
      build/debug/mongo/s/catalog/libsharding_catalog_client.so $
      build/debug/third_party/boost-1.70.0/libboost_program_options.so $
      build/debug/mongo/db/matcher/matcher_type_set.dyn.o build/debug/mongo/db/geo/libgeometry.so $
      build/debug/mongo/stdx/libstdx.so build/debug/mongo/db/matcher/rewrite_expr.dyn.o $
      build/debug/third_party/boost-1.70.0/libboost_log.so $
      build/debug/mongo/db/libkeys_collection_client_sharded.so $
      build/debug/mongo/db/exec/document_value/libdocument_value.so $
      build/debug/mongo/db/pipeline/libdocument_sources_idl.so $
      build/debug/mongo/crypto/libsha1_block.so $
      build/debug/mongo/rpc/libmetadata_impersonated_user.so $
      build/debug/third_party/boost-1.70.0/libboost_thread.so $
      build/debug/third_party/boost-1.70.0/libboost_iostreams.so $
      build/debug/mongo/db/query/libhint_parser.so $
      build/debug/third_party/murmurhash3/libmurmurhash3.so $
      build/debug/third_party/s2/base/libbase_s2.so $
      build/debug/mongo/db/matcher/schema/expression_internal_schema_object_match.dyn.o $
      build/debug/mongo/db/matcher/schema/encrypt_schema_types.dyn.o $
      build/debug/mongo/rpc/libmetadata.so $
      build/debug/mongo/db/pipeline/libruntime_constants_idl.so $
      build/debug/mongo/db/libwrite_concern_options.so build/debug/mongo/util/libclock_sources.so $
      build/debug/third_party/abseil-cpp-master/libabsl_throw_delegate.so $
      build/debug/mongo/db/bson/libdotted_path_support.so $
      build/debug/mongo/db/matcher/schema/expression_internal_schema_num_array_items.dyn.o $
      build/debug/mongo/util/libregex_util.so $
      build/debug/third_party/boost-1.70.0/libboost_system.so $
      build/debug/mongo/db/pipeline/libexpression.so build/debug/mongo/db/storage/libkey_string.so $
      build/debug/mongo/db/query/libexplain_options.so build/debug/mongo/s/libis_mongos.so $
      build/debug/mongo/db/matcher/expression_text_noop.dyn.o $
      build/debug/mongo/db/matcher/expression_parser.dyn.o $
      build/debug/mongo/client/libread_preference.so $
      build/debug/mongo/db/query/collation/libcollator_interface.so $
      build/debug/mongo/db/matcher/expression_expr.dyn.o $
      build/debug/mongo/util/options_parser/liboptions_parser.so $
      build/debug/mongo/client/libconnection_string.so build/debug/mongo/db/libindex_names.so $
      build/debug/mongo/db/repl/libstorage_interface.so $
      build/debug/mongo/util/libsecure_zero_memory.so build/debug/mongo/db/libdbmessage.so $
      build/debug/third_party/libshim_fmt.so build/debug/mongo/util/net/libnetwork.so $
      build/debug/mongo/client/libclientdriver_minimal.so $
     build/debug/mongo/db/matcher/expression_geo.dyn.o $
     build/debug/mongo/db/matcher/schema/expression_internal_schema_cond.dyn.o $
     build/debug/mongo/db/matcher/schema/json_pointer.dyn.o $
103     build/debug/mongo/base/libsecure_allocator.so $
     build/debug/third_party/s2/util/coding/libcoding.so $
     build/debug/mongo/db/matcher/schema/expression_internal_schema_fmod.dyn.o $
     build/debug/mongo/db/catalog/libcollection.so $
     build/debug/mongo/db/matcher/schema/expression_internal_schema_unique_items.dyn.o $
     build/debug/mongo/db/libserver_options_core.so build/debug/mongo/util/libquick_exit.so $
     build/debug/mongo/db/matcher/schema/expression_internal_schema_xor.dyn.o $
     build/debug/third_party/libshim_timelib.so $
     build/debug/mongo/db/matcher/expression_where_base.dyn.o $
     build/debug/mongo/db/matcher/matchable.dyn.o $
     build/debug/mongo/db/query/datetime/libdate_time_support.so $
     build/debug/third_party/fmt/libfmt.so build/debug/mongo/db/libwire_version.so $
     build/debug/third_party/libshim_yaml.so $
     build/debug/mongo/transport/libtransport_layer_common.so $
     build/debug/mongo/shell/libmongojs.so build/debug/mongo/db/repl/libread_concern_args.so $
     build/debug/mongo/db/ops/libwrite_ops_parsers.so $
     build/debug/mongo/db/repl/librepl_coordinator_interface.so $
     build/debug/mongo/db/liblogical_time.so build/debug/mongo/util/libsummation.so $
     build/debug/mongo/db/repl/librepl_settings.so build/debug/mongo/util/libmd5.so $
     build/debug/mongo/db/pipeline/libfield_path.so $
     build/debug/mongo/db/auth/libaddress_restriction.so build/debug/mongo/rpc/libprotocol.so $
     build/debug/mongo/db/matcher/schema/json_schema_parser.dyn.o build/debug/mongo/libbase.so $
     build/debug/mongo/client/libauthentication.so $
     build/debug/mongo/db/pipeline/libdependencies.so $
     build/debug/mongo/db/libmulti_key_path_tracker.so $
     build/debug/mongo/scripting/libbson_template_evaluator.so $
     build/debug/mongo/db/query/libquery_request.so $
     build/debug/mongo/db/matcher/expression_internal_expr_eq.dyn.o $
     build/debug/mongo/db/query/collation/libcollator_factory_interface.so $
     build/debug/third_party/s2/util/math/libmath.so build/debug/mongo/util/libsafe_num.so $
     build/debug/mongo/db/commands/libserver_status_core.so build/debug/mongo/s/libcommon_s.so $
     build/debug/mongo/util/concurrency/libspin_lock.so $
     build/debug/mongo/crypto/libsha256_block.so build/debug/mongo/db/auth/libauth.so $
     build/debug/mongo/db/libcommand_generic_argument.so $
     build/debug/mongo/db/storage/librecovery_unit_base.so $
     build/debug/mongo/db/matcher/extensions_callback_noop.dyn.o $
     build/debug/mongo/scripting/libscripting_common.so build/debug/mongo/db/matcher/matcher.dyn.o $
     build/debug/mongo/util/libicu.so build/debug/mongo/db/libglobal_settings.so $
     build/debug/mongo/db/matcher/schema/expression_internal_schema_allowed_properties.dyn.o $
     build/debug/mongo/idl/libidl_parser.so $
     build/debug/mongo/db/repl/liboptime_and_wall_time_base.so $
     build/debug/mongo/db/liblogical_clock.so build/debug/mongo/db/matcher/libpath.so $
     build/debug/mongo/util/libprocessinfo.so $
     build/debug/third_party/IntelRDFPMathLib20U1/libintel_decimal128.so $
     build/debug/third_party/s2/strings/libstrings.so $
     build/debug/mongo/executor/libremote_command.so $
     build/debug/third_party/abseil-cpp-master/libabsl_container.so $
     build/debug/mongo/libboost_assert_shim.so build/debug/mongo/util/libperiodic_runner.so $
     build/debug/mongo/db/libsigned_logical_time.so build/debug/third_party/s2/libs2.so $
     build/debug/mongo/idl/libserver_parameter.so $
153     build/debug/mongo/db/pipeline/libaggregation_request.so $
     build/debug/mongo/db/catalog/libcollection_catalog.so build/debug/mongo/db/libcommon.so $
     build/debug/mongo/db/libkeys_collection_manager.so $
     build/debug/third_party/abseil-cpp-master/libabsl_hash.so $
     build/debug/third_party/boost-1.70.0/libboost_filesystem.so $
     build/debug/mongo/bson/mutable/libmutable_bson.so $
     build/debug/mongo/db/matcher/schema/expression_internal_schema_all_elem_match_from_index.dyn.o $
     build/debug/third_party/libshim_abseil.so $
     build/debug/mongo/db/matcher/schema/encrypt_schema_gen.dyn.o $
     build/debug/mongo/rpc/libcommand_status.so $
     build/debug/mongo/db/matcher/schema/expression_internal_schema_root_doc_eq.dyn.o $
     build/debug/mongo/db/matcher/expression_with_placeholder.dyn.o $
     build/debug/mongo/db/matcher/expression_where_noop.dyn.o $
     build/debug/mongo/db/matcher/match_details.dyn.o build/debug/mongo/db/repl/librollback_idl.so $
     build/debug/mongo/db/libservice_context.so build/debug/mongo/db/auth/libauthprivilege.so $
     build/debug/mongo/crypto/libsha_block_openssl.so $
     build/debug/mongo/db/query/libcommand_request_response.so $
     build/debug/mongo/db/libkeys_collection_document.so build/debug/mongo/db/repl/liboptime.so $
     build/debug/mongo/db/liblogical_session_id.so $
     build/debug/mongo/db/commands/libtest_commands_enabled.so $
     build/debug/mongo/db/matcher/schema/expression_internal_schema_num_properties.dyn.o $
     build/debug/mongo/bson/util/libbson_extract.so $
     build/debug/third_party/libshim_intel_decimal128.so $
     build/debug/mongo/db/pipeline/libexpression_context.so $
     build/debug/mongo/db/matcher/expression_array.dyn.o $
     build/debug/mongo/db/matcher/expression_tree.dyn.o build/debug/third_party/libshim_pcrecpp.so $
     build/debug/third_party/yaml-cpp-0.6.2/libyaml.so build/debug/third_party/libshim_boost.so $
     build/debug/mongo/client/libsasl_iam_client.so build/debug/mongo/db/matcher/expression.dyn.o $
     /home/chasinglogic/Work/mongo/build.ninja || _generated_sources
   cmd = PATH=/opt/mongodbtoolchain/v3/bin:/usr/local/bin:/opt/bin:/bin:/usr/bin CCACHE_NOCPP2=1 $
       ICECC_VERSION=/data/build/scons/icecc/3bd3951bba3f1291547efbcc21098663.tar.gz $
       CCACHE_PREFIX=/bin/icecc /bin/icerun /opt/mongodbtoolchain/v3/bin/g++ $
       @/home/chasinglogic/Work/mongo/build/debug/response_files/tmpbx3uaji3.lnk

we can see that it's not a dependency wiring problem because the so's that are failing to be found are listed as implicit dependencies. For the problem we're seeing here there is no noticeable difference between explicit and implicit dependencies per the ninja build manual:

There are three types of build dependencies which are subtly different.
    
   - Explicit dependencies, as listed in a build line. These are available as the $in variable in the rule. Changes in these files cause the output to be rebuilt; if these file are missing and Ninja doesn't know how to build them, the build is aborted.
    This is the standard form of dependency to be used e.g. for the source file of a compile command.
    
  - Implicit dependencies, either as picked up from a depfile attribute on a rule or from the syntax | dep1 dep2 on the end of a build line. The semantics are identical to explicit dependencies, the only difference is that implicit dependencies don't show up in the $in variable. 

Generated at Thu Feb 08 05:09:52 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.