[SERVER-45990] Ninja should not have to rebuild everything when build.ninja file is modified Created: 05/Feb/20 Updated: 06/Dec/22 Resolved: 05/Feb/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Louis Williams | Assignee: | [DO NOT ASSIGN] Backlog - Server Development Platform Team (SDP) (Inactive) |
| Resolution: | Won't Fix | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Assigned Teams: |
Server Development Platform
|
| Participants: |
| Description |
|
If my build.ninja file is modified at all, then ninja must perform a full rebuild of all 3k+ objects for the mongod binary. This occurs most during git branch checkouts, which often modifies Scons dependencies. This makes branch checkouts, a commonly recommended server workflow, impractical. The old ninja module was able to determine which files actually required rebuilding. We should consider making a similar optimization for the new Ninja generator. |
| Comments |
| Comment by Mathew Robinson (Inactive) [ 05/Feb/20 ] |
|
I'm pretty sure this cannot be true louis.williams and at least if it was that was a bug in the old module. Here is where the generator rule for the build.ninja was created in the old module: https://github.com/RedBeard0531/mongo_module_ninja/blob/master/build.py#L1238 Which is the same as our rule except we don't track the dependencies in a depsfile and instead put them in the ninja file itself. This does explain why the old module was resilient to SConscript removals and we aren't though as per the ninja build manual depsfiles are resilient to files listed there being removed. Here is where the write method is defined you can see that it unconditionally rewrites the ninja file: https://github.com/RedBeard0531/mongo_module_ninja/blob/master/build.py#L1023 Here is where that method is called so there is no conditionals here: https://github.com/RedBeard0531/mongo_module_ninja/blob/master/build.py#L73 We did write a version of the new ninja generator which did this but it comes at a steep performance cost because you have to ensure all dependency, target, and build order listings are deterministic. (read: a lot of list sorting in loops) and chose to leave that out since with Icecream and ccache a full rebuild can happen very quickly. If you can set up this scenario with the old module on your machine where it doesn't rebuild the world when swapping branches I'd be very interested to come see it so I can see how we can implement it on our side. But we conversed with some other ninja users in the office and it seems they always experienced full rebuilds after a regeneration. Additionally other build systems that specialize in ninja generation like Meson also work this way with explicit listings of build.ninja implicit dependencies and requiring full rebuilds after a ninja regeneration. Here is an in-depth explanation of why it works this way: All build targets in the ninja file depend on the Ninja file. This is for two reasons. One so that the ninja file is checked if it needs to be regenerated first thing in the build and two so that if it is regenerated all targets are rebuilt. Ninja works entirely off of timestamps to determine out-of-dateness as opposed to content signatures used by other build systems, like SCons. This has the benefit of being very fast, but at the cost of being simple. It means that we cannot do things like make a target depend only on it's CFLAGS. So if for instance we made the build.ninja file an order-only dependency of everything (so it would still be checked first thing in a build but not cause a full-rebuild) and we changed the SConstruct so that it adds a new required CFLAG to our build the ninja file would regenerate but no targets would rebuild with the new CFLAG. Because Ninja only operates on timestamps you never have the ability to tie that change of command line to the resulting target. So all ninja generators must enforce that all targets depend on the ninja file and rebuild if it changes. |