[SERVER-79790] Establish mechanism to produce dependency graph Created: 07/Aug/23  Updated: 29/Oct/23  Resolved: 13/Sep/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.2.0-rc0

Type: New Feature Priority: Major - P3
Reporter: Steve Gross Assignee: Zack Winter
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Backwards Compatibility: Fully Compatible
Participants:

 Description   

As part of the migrate-target-to-Bazel process, we need to pick off "leaf nodes" from the build graph. To do so, we need a mechanism to analyze and visualize the graph itself.



 Comments   
Comment by Githook User [ 20/Sep/23 ]

Author:

{'name': 'Zack Winter', 'email': 'zack.winter@mongodb.com', 'username': 'zackwintermdb'}

Message: SERVER-79790 Update libdeps readme to include arg to skip compiling enterprise module
Branch: master
https://github.com/mongodb/mongo/commit/46ad24e0cecb40e31fb9f7b74d91fb601144cc35

Comment by Zack Winter [ 13/Sep/23 ]

The following snippet now generates a list of leaf nodes without non-bazelified dependencies that are ready for bazel conversion:

python3 buildscripts/libdeps/gacli.py --graph-file build/opt/libdeps/libdeps.graphml --bazel-conv-candidates 

Comment by Githook User [ 13/Sep/23 ]

Author:

{'name': 'Zack Winter', 'email': 'zack.winter@mongodb.com', 'username': 'zackwintermdb'}

Message: SERVER-79790 Add option to generate bazel conversion targets to libdeps
Branch: master
https://github.com/mongodb/mongo/commit/7cccfc0caf6e3b049744aff498bb91ad28113df6

Comment by Steve Gross [ 22/Aug/23 ]

Note: assigned this to zack.winter@mongodb.com . He's got a few things on the docket right now but this makes sense to do when those things are done.

Comment by Steve Gross [ 08/Aug/23 ]

alex.neben@mongodb.com Thanks, I'll take a look. The list in the scope is just an initial list, however. I'll see if the libdeps visualizer can give us a comprehensive picture. (Unfortunately, #include-based dependencies don't seem to show up in our SCons build, but they are enforced in a Bazel build, so this may get complicated!).

Comment by Alex Neben [ 07/Aug/23 ]

We already have this information as part of "Choosing good candidate libraries for early Bazel work" section in the scope. We have some guides for using the libdeps visualizer which I would take a crack at before trying to roll your own. I think that can be found "buildscripts/libdeps/README.md"

Comment by Steve Gross [ 07/Aug/23 ]

Note to self: another promising technique is creating a dependency graph based entirely on #includes. Currently using this command: grep -E "#include" `find src/mongo -type f | grep -E '\.h$'` | grep -v -E '>$' | sed 's/\//__/g' | sed 's/:#include "/ -> /g' | sed 's/"$//g' | sed 's/\.h//g' | sort -u

Comment by Steve Gross [ 07/Aug/23 ]

Note to self: I'm working on filtering out the source files themselves from the tree (cat /tmp/scons_tree.txt | grep -v -E "\.(hpp|py|idl|h|cpp|yml|inc|tpl|o|in|ipp|hh|c|def|defs|js|txt)$" | grep -v -E "/bin/(ar|g++|gcc)$" | grep -v -E "src/third_party" | more)

Comment by Steve Gross [ 07/Aug/23 ]

Note to self: I can produce an initial build graph using python3 ./buildscripts/scons.py --tree=all --linker=gold 2>&. Now I'm working on translating that into dot notation so we can understand the graph better.

Generated at Thu Feb 08 06:41:54 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.