[SERVER-64664] ninja tool should not consider install files generated source Created: 18/Mar/22  Updated: 29/Oct/23  Resolved: 13/Apr/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 5.3.2, 5.0.9, 4.4.15

Type: Task Priority: Major - P3
Reporter: Daniel Moody Assignee: Daniel Moody
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Duplicate
is duplicated by SERVER-44731 Make definition of generated sources ... Closed
Related
is related to SERVER-54914 Add Copy function action handler to n... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v5.3, v5.0, v4.4
Sprint: Dev Platform 2022-04-04, Dev Platform 2022-04-18
Participants:

 Description   

The ninja tool looks at any output with a header file as generated source. However install edges have outputs with header files, and they get labeled as generated source. This can cause circular dependencies in a few edge cases, and should be a general improvement.

Possible fix is just checking for the "INSTALL" rule when determining if a file is generated source:

diff --git a/site_scons/site_tools/ninja.py b/site_scons/site_tools/ninja.py
index 5826be827f4..d621bb3ada0 100644
--- a/site_scons/site_tools/ninja.py
+++ b/site_scons/site_tools/ninja.py
@@ -664,7 +664,7 @@ class NinjaState:
             output
             # First find builds which have header files in their outputs.
             for build in self.builds.values()
-            if self.has_generated_sources(build["outputs"])
+            if build['rule'] != 'INSTALL' and self.has_generated_sources(build["outputs"])
             for output in build["outputs"]
             # Collect only the header files from the builds with them
             # in their output. We do this because is_generated_source



 Comments   
Comment by Githook User [ 26/Apr/22 ]

Author:

{'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}

Message: SERVER-64664 add alias tagging for ninja to determine generated sources.

(cherry picked from commit 2fef432fa6e7cf3fd4f22ba3b193222c2887f14f)
Branch: v5.3
https://github.com/mongodb/mongo/commit/8467a3225b586a6c35391e169ee5fc4c5c552600

Comment by Githook User [ 25/Apr/22 ]

Author:

{'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}

Message: SERVER-64664 add alias tagging for ninja to determine generated sources.

(cherry picked from commit 2fef432fa6e7cf3fd4f22ba3b193222c2887f14f)
Branch: v5.0
https://github.com/mongodb/mongo/commit/3539833b79dea8787dc7e44619c6776126e81a55

Comment by Githook User [ 25/Apr/22 ]

Author:

{'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}

Message: SERVER-64664 add alias tagging for ninja to determine generated sources.

(cherry picked from commit 2fef432fa6e7cf3fd4f22ba3b193222c2887f14f)
Branch: v4.4
https://github.com/mongodb/mongo/commit/6c8521729466a841127e62e54bbbaaaebbf35330

Comment by Githook User [ 11/Apr/22 ]

Author:

{'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}

Message: SERVER-64664 add alias tagging for ninja to determine generated sources.
Branch: master
https://github.com/mongodb/mongo/commit/2fef432fa6e7cf3fd4f22ba3b193222c2887f14f

Comment by Andrew Morrow (Inactive) [ 19/Mar/22 ]

I like this as a workaround, and I think richard.samuels can just implement it as part of his work on SERVER-62992. I'm curious whether we could come up with a rule that allowed us to automatically identify generated sources. It has always been a weakness of the current strategy that we need to manually tag generated sources in our SCons implementation by adding them to the generated-sources Alias. Maybe if we spend a little time thinking carefully about what makes a source be "generated", we can solve several problems at once: eliminating the need for the manual tagging in the SCons layer, and eliminating the need for special handing in the Ninja layer. It would maybe also be useful later in a SConserver environment so we could just callback to SCons as needed when we hit a dependency on a generated source?

So, what makes a source "generated"? How do we distinguish them from other generated things, like object files, which also become in some sense sources, but are not generated in the same way as the things we currently consider "generated sources".

Generated at Thu Feb 08 06:00:52 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.