[SERVER-54285] "Internal Error" message spam when printing libdep cycles Created: 04/Feb/21  Updated: 29/Oct/23  Resolved: 08/Mar/21

Status: Closed
Project: Core Server
Component/s: Build
Affects Version/s: None
Fix Version/s: 4.9.0

Type: Bug Priority: Major - P3
Reporter: Mathias Stearn Assignee: Andrew Morrow (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.4
Steps To Reproduce:

Build command: /usr/bin/python3 "./buildscripts/scons.py" "VARIANT_DIR=ninja" "--variables-files=etc/scons/mongodbtoolchain_stable_gcc.vars" "--ninja" "generate-ninja" "ICECC=icecc" "CCACHE=ccache" "--link-model=dynamic" "--dbg" "--opt" "--linker=gold" "CCFLAGS=-gsplit-dwarf -fdiagnostics-color"

Sprint: Dev Platform 2021-02-22, Dev Platform 2021-03-08
Participants:

 Description   

  Internal Error: no cycle found for node install-all-meta (<SCons.Node.Alias.Alias object at 0x7f7e930f4048>) in state pending
  Internal Error: no cycle found for node build/ninja/mongo/db/commands/libshell_protocol.so (<SCons.Node.FS.File object at 0x4068058>) in state pending
  Internal Error: no cycle found for node install-all (<SCons.Node.Alias.Alias object at 0x7f7e930e8048>) in state pending
  Internal Error: no cycle found for node build/ninja/mongo/db/query/db_query_test (<SCons.Node.FS.File object at 0x4430048>) in state pending
  Internal Error: no cycle found for node build/install/bin/db_commands_test (<SCons.Node.FS.File object at 0x40b0098>) in state pending
  Internal Error: no cycle found for node build/ninja/pkgs/benchmarks-meta.tgz (<SCons.Node.FS.File object at 0x4ec00a8>) in state pending
  Internal Error: no cycle found for node build/ninja/mongo/db/db_unittest_with_config_server_test_fixture_test (<SCons.Node.FS.File object at 0x49440b8>) in state pending
  Internal Error: no cycle found for node build/install/lib/libviews_mongod.so (<SCons.Node.FS.File object at 0x476c0c8>) in state pending
  Internal Error: no cycle found for node build/ninja/mongo/db/storage/wiredtiger/libstorage_wiredtiger_core.so (<SCons.Node.FS.File object at 0x471c0f8>) in state pending
  Internal Error: no cycle found for node build/ninja/mongo/db/catalog/libhealth_log.so (<SCons.Node.FS.File object at 0x3f74128>) in state pending
  Internal Error: no cycle found for node build/install/lib/libindex_build_entry_helpers.so (<SCons.Node.FS.File object at 0x4840158>) in state pending
  Internal Error: no cycle found for node build/ninja/pkgs/dist-test-dev.tgz (<SCons.Node.FS.File object at 0x4eb8158>) in state pending
  Internal Error: no cycle found for node build/ninja/pkgs/embedded-runtime.tgz (<SCons.Node.FS.File object at 0x36d8158>) in state pending
  Internal Error: no cycle found for node build/ninja/mongo/db/catalog/libcatalog_control.so (<SCons.Node.FS.File object at 0x3fb8198>) in state pending
  Internal Error: no cycle found for node build/install/lib/libhealth_log.so (<SCons.Node.FS.File object at 0x3f881d8>) in state pending
  Internal Error: no cycle found for node install-dist-test-meta (<SCons.Node.Alias.Alias object at 0x7f7e930f41d0>) in state pending
  Internal Error: no cycle found for node build/ninja/mongo/db/storage/wiredtiger/libstorage_wiredtiger.so (<SCons.Node.FS.File object at 0x3fec208>) in state pending
  Internal Error: no cycle found for node build/install/bin/db_storage_test (<SCons.Node.FS.File object at 0x4778228>) in state pending
  build/install/lib/libindex_access_method.so -> build/install/lib/libduplicate_key_tracker.so -> build/install/lib/libindex_access_method.so
  Internal Error: no cycle found for node build/ninja/mongo/db/libwrite_ops.so (<SCons.Node.FS.File object at 0x3fd0278>) in state pending
  Internal Error: no cycle found for node build/ninja/mongo/db/repl/libinitial_syncer.so (<SCons.Node.FS.File object at 0x448c278>) in state pending
  Internal Error: no cycle found for node build/install/bin/db_query_test (<SCons.Node.FS.File object at 0x44302a8>) in state pending
  Internal Error: no cycle found for node build/ninja/mongo/db/repl/librollback_impl.so (<SCons.Node.FS.File object at 0x44682b8>) in state pending
  Internal Error: no cycle found for node build/ninja/pkgs/mongod-dev.tgz (<SCons.Node.FS.File object at 0x4ecc2b8>) in state pending
  Internal Error: no cycle found for node build/install/lib/libcollection_validation.so (<SCons.Node.FS.File object at 0x3f982d8>) in state pending
  Internal Error: no cycle found for node build/install/lib/libperiodic_runner_job_abort_expired_transactions.so (<SCons.Node.FS.File object at 0x48c42f8>) in state pending
  Internal Error: no cycle found for node build/ninja/mongo/db/repl/libinitial_sync_cloners.so (<SCons.Node.FS.File object at 0x4098358>) in state pending
  Internal Error: no cycle found for node build/install/lib/libstartup_recovery.so (<SCons.Node.FS.File object at 0x489c368>) in state pending
  Internal Error: no cycle found for node build/install/lib/libstorage_wiredtiger.so (<SCons.Node.FS.File object at 0x46cc378>) in state pending
  Internal Error: no cycle found for node build/ninja/mongo/db/concurrency/libdeferred_writer.so (<SCons.Node.FS.File object at 0x3f74388>) in state pending
  Internal Error: no cycle found for node build/ninja/mongo/db/storage/devnull/libstorage_devnull.so (<SCons.Node.FS.File object at 0x46e4358>) in state pending
  Internal Error: no cycle found for node install-servers (<SCons.Node.Alias.Alias object at 0x7f7e930e8358>) in state pending
  Internal Error: no cycle found for node install-dbtest-meta (<SCons.Node.Alias.Alias object at 0x7f7e930f4358>) in state pending
  Internal Error: no cycle found for node build/install/bin/storage_wiredtiger_test (<SCons.Node.FS.File object at 0x47283e8>) in state pending
  Internal Error: no cycle found for node build/ninja/mongo/db/libindex_builds_coordinator_interface.so (<SCons.Node.FS.File object at 0x3fb83f8>) in state pending
  Internal Error: no cycle found for node build/ninja/mongo/db/repl/librepl_coordinator_test_fixture.so (<SCons.Node.FS.File object at 0x4484408>) in state pending
  Internal Error: no cycle found for node build/ninja/mongo/db/repl/librollback_test_fixture.so (<SCons.Node.FS.File object at 0x4498458>) in state pending
  Internal Error: no cycle found for node build/install/lib/libmongod.so (<SCons.Node.FS.File object at 0x408c488>) in state pending
  Internal Error: no cycle found for node build/install/lib/librepl_coordinator_test_fixture.so (<SCons.Node.FS.File object at 0x44b0498>) in state pending
  Internal Error: no cycle found for node build/ninja/mongo/db/s/libsharding_mongod_test_fixture.so (<SCons.Node.FS.File object at 0x45604a8>) in state pending
  Internal Error: no cycle found for node build/ninja/mongo/db/catalog/libcatalog_impl.so (<SCons.Node.FS.File object at 0x3fc04c8>) in state pending
  Internal Error: no cycle found for node build/install/lib/liboplog_interface_local.so (<SCons.Node.FS.File object at 0x44884f8>) in state pending
  Internal Error: no cycle found for node build/ninja/mongo/db/repl/librs_rollback.so (<SCons.Node.FS.File object at 0x4468518>) in state pending
  Internal Error: no cycle found for node build/ninja/pkgs/dist-meta.tgz (<SCons.Node.FS.File object at 0x4ed0538>) in state pending
  Internal Error: no cycle found for node build/ninja/mongo/db/repl/db_repl_cloners_test (<SCons.Node.FS.File object at 0x45685e8>) in state pending
  Internal Error: no cycle found for node build/ninja/mongo/db/repl/liboplog_application.so (<SCons.Node.FS.File object at 0x44785f8>) in state pending
  Internal Error: no cycle found for node build/ninja/pkgs/embedded-dev.tgz (<SCons.Node.FS.File object at 0x36d8618>) in state pending

This is just a sample, it goes on for many pages of this.



 Comments   
Comment by Githook User [ 05/Mar/21 ]

Author:

{'name': 'Andrew Morrow', 'email': 'acm@mongodb.com', 'username': 'acmorrow'}

Message: SERVER-54285 Ensure cycle detection traverses private edges
Branch: master
https://github.com/mongodb/mongo/commit/3dcd089735f024a0fffcfdff446215492f9281e6

Comment by Daniel Moody [ 26/Feb/21 ]

We have decided to just bring over the singular fix for now, as opposed promoting next to stable because we have several issues we want to complete on next, and wait for SCons 4.1 update.

Comment by Daniel Moody [ 25/Feb/21 ]

The excessive printing around dependency cycles appears to be fixed with --build-tools=next option.

Comment by Daniel Moody [ 04/Feb/21 ]

Not much we can do as user of SCons since this is a SCons internal issue. I suppose we could catch the exception and strip out the extraneous info before re-emmiting? There are probably real no cycle cases though that you want that information.

Comment by Daniel Moody [ 04/Feb/21 ]

Its not ninja specific.

The no cycle means that some nodes never finished building, which can happen when there is a dependency cycle because SCons will never build the nodes that have the actual dependency cycle, so any nodes that depend on those never get built, and SCons runs out of tasks it can start on and finishes.

Its extra information about which nodes were affected by your dependency cycle, but it is not very useful.

Comment by Andrew Morrow (Inactive) [ 04/Feb/21 ]

Would be good to know if it does or does not repro without --ninja but with the deliberate cycle. The Ninja tool is pretty invasive, it is definitely possible that it has caused the taskmaster state to be in a bad place such that when it tries to do its cycle reporting, the expected information is just not there. We might need to override that behavior when Ninja is in play.

Comment by Mathias Stearn [ 04/Feb/21 ]

And this seems to be the minimal set of flags to repro with: python3 buildscripts/scons.py --variables-files=etc/scons/mongodbtoolchain_stable_gcc.vars --link-model=dynamic --ninja

Comment by Mathias Stearn [ 04/Feb/21 ]

Weirdly it seems to only do it for some cycles. Here is a patch that causes it on top of clean master (efbacd78d539c761e3af82e66f0e7ac7c9247d65):

diff --git a/src/mongo/db/index/SConscript b/src/mongo/db/index/SConscript
index 8cad299239..d3927f8f6c 100644
--- a/src/mongo/db/index/SConscript
+++ b/src/mongo/db/index/SConscript
@@ -35,6 +35,7 @@ env.Library(
             '$BUILD_DIR/mongo/db/curop',
             '$BUILD_DIR/mongo/db/storage/execution_context',
             '$BUILD_DIR/mongo/db/storage/index_entry_comparison',
+            'index_build_interceptor'
         ],
 )

Comment by Andrew Morrow (Inactive) [ 04/Feb/21 ]

Yeah, I've seen this happen before too when a cycle really existed. I agree it would be nice if it didn't happen; the error is definitely confusing.

Comment by Mathias Stearn [ 04/Feb/21 ]

I am forked off at 8baad6fdb799b24bbc999089773698a10bfeaecc. This is on a virtual workstation if that matters. I'll see if I can repro with a clean checkout modified only to have a cycle.

Comment by Daniel Moody [ 04/Feb/21 ]

I could not reproduce on master (4abad9089912ff62e492923cafd2315412f6600b), what commit are you on?

Generated at Thu Feb 08 05:33:06 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.