[SERVER-48289] New ninja causes icecream to run jobs locally with clang Created: 19/May/20 Updated: 29/Oct/23 Resolved: 29/May/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Build |
| Affects Version/s: | None |
| Fix Version/s: | 4.4.0-rc8, 4.7.0 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Randolph Tan | Assignee: | Ryan Egesdahl (Inactive) |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||
| Issue Links: |
|
||||
| Backwards Compatibility: | Fully Compatible | ||||
| Operating System: | ALL | ||||
| Backport Requested: |
v4.4
|
||||
| Sprint: | Dev Platform 2020-06-01 | ||||
| Participants: | |||||
| Description |
|
icecc 1.1 Command used to create ninja file:
Ninja file generated: However, when I switched to gcc, icecream works as expected
|
| Comments |
| Comment by Githook User [ 01/Jun/20 ] | |||||||||||
|
Author: {'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}Message: Some compiler flags passed in from the environment can cause icecc to (cherry picked from commit 1248eaec6e4d73484af9c2220ea9d22730ecbe44) | |||||||||||
| Comment by Githook User [ 29/May/20 ] | |||||||||||
|
Author: {'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}Message: Some compiler flags passed in from the environment can cause icecc to | |||||||||||
| Comment by Ryan Egesdahl (Inactive) [ 28/May/20 ] | |||||||||||
|
I think this is going to be fairly common for people who are moving forward with the new Ninja build system, so I'm going to add -Qunused-arguments to the new Ninja system to prevent confusion. I'll do that tonight, and it should hopefully be merged by the end of the week. For future reference of those reading this ticket: adding your own compiler flags can be somewhat risky business in complex build systems such as ours. Performance flags especially tend to fall in the "set it because someone said so and forget it" category (I've done this, too), and they're going to be a common trap as our build system continues to evolve. If you don't have a clear need for a flag (meaning you know exactly what it does and that you need it to happen), don't add it to your build until you know what it does and that you really do need it. If you have any flags currently set, examine each one, ask an expert if you don't know what it does, and remove it if you don't absolutely need it. | |||||||||||
| Comment by Randolph Tan [ 27/May/20 ] | |||||||||||
|
Looks like icecream is now working with clang as expected after removing the flags. | |||||||||||
| Comment by Ryan Egesdahl (Inactive) [ 26/May/20 ] | |||||||||||
|
Did you just recently move to --ninja=next as of when this began to fail? I ask because clang++ will default to using llvm-as on systems that support it - which is basically any Linux or recent OSX - and llvm-as doesn't understand --compress-debug-sections at all. In LLVM, it's the linker (ld.lld) which takes that flag. As it happens, the old ninja module was passing -Qunused-arguments to clang (and only clang) to get around some problems where flags were being passed in externally that clang didn't understand. Try removing --Wa,--compress-debug-sections from your flags and see if it fixes the problem for you. | |||||||||||
| Comment by Ryan Egesdahl (Inactive) [ 26/May/20 ] | |||||||||||
|
Nevermind about not being able to reproduce - that's likely because you're using a different icecc version than I am. I expect that the mechanism is exactly what we think it is: the new ninja isn't using -Qunused-arguments. I'll head down that path now. | |||||||||||
| Comment by Ryan Egesdahl (Inactive) [ 26/May/20 ] | |||||||||||
|
I was still completely unable to reproduce what you are seeing, and I am starting to suspect toolchain+icecc shenanigans here. Could you please try restarting iceccd and clearing ccache, then trying exactly the same build again? That will clear the cached toolchain that gets sent to the icecream cluster so we can eliminate it as a potential cause. | |||||||||||
| Comment by Randolph Tan [ 26/May/20 ] | |||||||||||
|
mongo_custom_variables.py
No env variables set for SCONSFLAG. Running on Fedora 29. | |||||||||||
| Comment by Ryan Egesdahl (Inactive) [ 26/May/20 ] | |||||||||||
|
I am not getting the same behavior when I run your exact scons commands, and the `ice.ninja` I get is significantly different from yours . Could you please give me the contents of your ~/.scons/site_scons/mongo_custom_variables.py and your environment variables? I suspect that the mechanism of what is happening here is probably obvious from the details, but I want to see what's triggering it. | |||||||||||
| Comment by Andrew Morrow (Inactive) [ 20/May/20 ] | |||||||||||
|
daniel.gottlieb - That is very possible. I was planning to look into it later today, but I'll start with that. | |||||||||||
| Comment by Daniel Gottlieb (Inactive) [ 20/May/20 ] | |||||||||||
|
Is this the issue where new ninja doesn't use -Qunused-arguments? I hadn't looked to see if that was limited to clang vs gcc. The ccache lookup (which doesn't compile, just preprocessing) produces an error because of the debug flags that affect compiling are unused for preprocessing. I don't recall looking at icecream logs when I had that problem, but I definitely observed the compilation happening locally. The documentation here describes how to use CCACHE_LOG to get better visibility into what CCACHE is doing. | |||||||||||
| Comment by Randolph Tan [ 19/May/20 ] | |||||||||||
|
Also attached icecc.log. The beginning of the logs are run with the ninja for clang and the logs starting around 18:21 are run with gcc. |