[SERVER-48289] New ninja causes icecream to run jobs locally with clang Created: 19/May/20  Updated: 29/Oct/23  Resolved: 29/May/20

Status: Closed
Project: Core Server
Component/s: Build
Affects Version/s: None
Fix Version/s: 4.4.0-rc8, 4.7.0

Type: Bug Priority: Major - P3
Reporter: Randolph Tan Assignee: Ryan Egesdahl (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File icecc.log    
Issue Links:
Backports
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.4
Sprint: Dev Platform 2020-06-01
Participants:

 Description   

icecc 1.1
ccache 3.4.2

Command used to create ninja file:

buildscripts/scons.py --variables-files=etc/scons/mongodbtoolchain_stable_clang.vars CCFLAGS='-Wa,--compress-debug-sections -gsplit-dwarf' VARIANT_DIR=ninja_ice --ninja generate-ninja NINJA_PREFIX=ice ICECC=icecc CCACHE=ccache --ninja=next

Ninja file generated:
https://gist.github.com/renctan/345790ec7f9a2cf648392aab434cf9a7

However, when I switched to gcc, icecream works as expected

buildscripts/scons.py --variables-files=etc/scons/mongodbtoolchain_stable_gcc.vars CCFLAGS='-Wa,--compress-debug-sections -gsplit-dwarf' VARIANT_DIR=ninja_ice --ninja generate-ninja NINJA_PREFIX=ice ICECC=icecc CCACHE=ccache --ninja=next



 Comments   
Comment by Githook User [ 01/Jun/20 ]

Author:

{'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}

Message: SERVER-48289 Ignore unused compiler flags

Some compiler flags passed in from the environment can cause icecc to
only build locally because the compiler gives an error saying the flag
is not supported. It may also have been causing problems with using
ccache in general. There may be other negative behaviors that we don't
know of that haven't been reported yet. This wasn't a problem in the
older Ninja build system, but it is in the newer one.

(cherry picked from commit 1248eaec6e4d73484af9c2220ea9d22730ecbe44)
Branch: v4.4
https://github.com/mongodb/mongo/commit/be44b28afa8503ad0c88c5db9f3e08fef8a30c46

Comment by Githook User [ 29/May/20 ]

Author:

{'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}

Message: SERVER-48289 Ignore unused compiler flags

Some compiler flags passed in from the environment can cause icecc to
only build locally because the compiler gives an error saying the flag
is not supported. It may also have been causing problems with using
ccache in general. There may be other negative behaviors that we don't
know of that haven't been reported yet. This wasn't a problem in the
older Ninja build system, but it is in the newer one.
Branch: master
https://github.com/mongodb/mongo/commit/1248eaec6e4d73484af9c2220ea9d22730ecbe44

Comment by Ryan Egesdahl (Inactive) [ 28/May/20 ]

I think this is going to be fairly common for people who are moving forward with the new Ninja build system, so I'm going to add -Qunused-arguments to the new Ninja system to prevent confusion. I'll do that tonight, and it should hopefully be merged by the end of the week.

For future reference of those reading this ticket: adding your own compiler flags can be somewhat risky business in complex build systems such as ours. Performance flags especially tend to fall in the "set it because someone said so and forget it" category (I've done this, too), and they're going to be a common trap as our build system continues to evolve. If you don't have a clear need for a flag (meaning you know exactly what it does and that you need it to happen), don't add it to your build until you know what it does and that you really do need it. If you have any flags currently set, examine each one, ask an expert if you don't know what it does, and remove it if you don't absolutely need it.

Comment by Randolph Tan [ 27/May/20 ]

Looks like icecream is now working with clang as expected after removing the flags.

Comment by Ryan Egesdahl (Inactive) [ 26/May/20 ]

Did you just recently move to --ninja=next as of when this began to fail? I ask because clang++ will default to using llvm-as on systems that support it - which is basically any Linux or recent OSX - and llvm-as doesn't understand --compress-debug-sections at all. In LLVM, it's the linker (ld.lld) which takes that flag.

As it happens, the old ninja module was passing -Qunused-arguments to clang (and only clang) to get around some problems where flags were being passed in externally that clang didn't understand. Try removing --Wa,--compress-debug-sections from your flags and see if it fixes the problem for you.

Comment by Ryan Egesdahl (Inactive) [ 26/May/20 ]

Nevermind about not being able to reproduce - that's likely because you're using a different icecc version than I am. I expect that the mechanism is exactly what we think it is: the new ninja isn't using -Qunused-arguments. I'll head down that path now.

Comment by Ryan Egesdahl (Inactive) [ 26/May/20 ]

I was still completely unable to reproduce what you are seeing, and I am starting to suspect toolchain+icecc shenanigans here. Could you please try restarting iceccd and clearing ccache, then trying exactly the same build again? That will clear the cached toolchain that gets sent to the icecream cluster so we can eliminate it as a potential cause.

Comment by Randolph Tan [ 26/May/20 ]

mongo_custom_variables.py

def short_describe():
    import os
    import subprocess
    with open(os.devnull, "r+") as devnull:
        proc = subprocess.Popen("git describe --abbrev=0",
            stdout=subprocess.PIPE,
            stderr=devnull,
            stdin=devnull,
            shell=True)
        return proc.communicate()[0].decode('utf-8').strip()[1:]MONGO_GIT_HASH="unknown"
MONGO_VERSION=short_describe()

No env variables set for SCONSFLAG. Running on Fedora 29.

Comment by Ryan Egesdahl (Inactive) [ 26/May/20 ]

I am not getting the same behavior when I run your exact scons commands, and the `ice.ninja` I get is significantly different from yours . Could you please give me the contents of your ~/.scons/site_scons/mongo_custom_variables.py and your environment variables? I suspect that the mechanism of what is happening here is probably obvious from the details, but I want to see what's triggering it.

Comment by Andrew Morrow (Inactive) [ 20/May/20 ]

daniel.gottlieb - That is very possible. I was planning to look into it later today, but I'll start with that.

Comment by Daniel Gottlieb (Inactive) [ 20/May/20 ]

Is this the issue where new ninja doesn't use -Qunused-arguments? I hadn't looked to see if that was limited to clang vs gcc. The ccache lookup (which doesn't compile, just preprocessing) produces an error because of the debug flags that affect compiling are unused for preprocessing. I don't recall looking at icecream logs when I had that problem, but I definitely observed the compilation happening locally.

The documentation here describes how to use CCACHE_LOG to get better visibility into what CCACHE is doing.

Comment by Randolph Tan [ 19/May/20 ]

Also attached icecc.log. The beginning of the logs are run with the ninja for clang and the logs starting around 18:21 are run with gcc.

Generated at Thu Feb 08 05:16:41 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.