[SERVER-56003] ninja + icecream with changing compilers doesn't regen run-icecc.sh Created: 09/Apr/21  Updated: 29/Oct/23  Resolved: 23/Mar/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 5.3.2, 4.4.14, 5.0.8, 5.3.1

Type: Bug Priority: Major - P3
Reporter: Daniel Moody Assignee: Daniel Moody
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Related
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v5.3, v5.0, v4.4
Steps To Reproduce:

as of 5ff2fd561d48c390a03e0d27239e3e378203634b

1. generate gcc ninja like so:

/opt/mongodbtoolchain/v3/bin/python3 buildscripts/scons.py --variables-files=etc/scons/mongodbtoolchain_v3_gcc.vars --ninja NINJA_PREFIX=gcc ICECC=icecc generate-ninja VARIANT_DIR=gcc

2. generate clang ninja like so:

/opt/mongodbtoolchain/v3/bin/python3 buildscripts/scons.py --variables-files=etc/scons/mongodbtoolchain_v3_clang.vars --ninja NINJA_PREFIX=clang ICECC=icecc generate-ninja VARIANT_DIR=clang

3. build clang, make sure to record the hash of the icecream compiler package just created

ninja -j400 -f clang.ninja install-mongo

4. build gcc, make sure to record the hash of the second compiler package just created

ninja -j400 -f gcc.ninja install-mongo

5. notice current run-icecc.sh is pointing to gcc compiler package
6. delete a file from the clang build:

rm -rf build/clang/mongo/client/mongo_uri.o

7. put on icecream debugging to see compiler fail to start:

export ICECC_DEBUG=icecc.log

8. run the clang build again, it is still point to the gcc icecream compiler package and will try to use that

ninja -j400 -f clang.ninja install-mongo

9. notice in the icecream debug logs:

ICECC[1550] 2022-01-26 11:27:58: flush_writebuf() failed(Error: Broken pipe)
ICECC[1550] 2022-01-26 11:27:58: setting error state for channel 10.122.7.201: (B eof: 0)
ICECC[1550] 2022-01-26 11:27:58: remote status: compiler did not start
ICECC[1550] 2022-01-26 11:27:58: write of source chunk to host 10.122.7.201
ICECC[1550] 2022-01-26 11:27:58: failed (Error: Broken pipe)
ICECC[1550] 2022-01-26 11:27:58: </write_fd_to_server from cpp: 370ms>
ICECC[1550] 2022-01-26 11:27:58: got exception Error 15 - write to host failed (10.122.7.201) 
ICECC[1550] 2022-01-26 11:27:58: connected to /var/run/icecc/iceccd.socket
ICECC[1550] 2022-01-26 11:27:58: <building_local>

Sprint: Dev Platform 2021-09-06, Dev Platform 2021-09-20, Dev Platform 2021-10-04, Dev Platform 2021-10-18, Dev Platform 2022-01-10, Dev Platform 2022-01-24, Dev Platform 2022-02-07, Dev Platform 2022-03-07, Dev Platform 2022-03-21, Dev Platform 2022-04-04
Participants:

 Description   

when generating multiple ninja files with different compilers in each, the run-icecc.sh script has no input file for ninja to check to see if it needs to be rebuilt.

Below is an example ninja edge of how run-icecc.sh is built:

build build/aib_make_archive.py build/cached/mongo/config.h $
    build/cached/mongo/util/version_constants.h build/cached/third_party/wiredtiger/wiredtiger.h $
    build/scons/icecream/run-icecc.sh: TEMPLATE src/mongo/config.h.in $
    src/mongo/util/version_constants.h.in src/third_party/wiredtiger/src/include/wiredtiger.in | $
    /usr/bin/icecc /opt/mongodbtoolchain/v3/bin/gcc /opt/mongodbtoolchain/v3/bin/g++

Here the compiler is an input to run-icecc.sh, but that is not enough because each ninja file will have a compiler with an unchanging mtime so run-icecc.sh will not be regenerated.

A proposed solution from redbeard0531 is to leverage the method from the ninja module: https://github.com/RedBeard0531/mongo_module_ninja/blob/master/build.py#L358-L385



 Comments   
Comment by Githook User [ 13/Apr/22 ]

Author:

{'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}

Message: SERVER-56003 use separate run icecc scripts for multiple ninja file builds.

(cherry picked from commit fc2b624564c1ec4f7d27c98e5a0074920e7ee5a1)
Branch: v5.3
https://github.com/mongodb/mongo/commit/fe670ef960a85d5fa01a58401529447d31dcd279

Comment by Githook User [ 13/Apr/22 ]

Author:

{'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}

Message: SERVER-56003 use separate run icecc scripts for multiple ninja file builds.

(cherry picked from commit fc2b624564c1ec4f7d27c98e5a0074920e7ee5a1)
Branch: v5.0
https://github.com/mongodb/mongo/commit/ce3523a5d5d57794ed7eadf7bf8631f4b4523ae9

Comment by Githook User [ 13/Apr/22 ]

Author:

{'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}

Message: SERVER-56003 use separate run icecc scripts for multiple ninja file builds.

(cherry picked from commit fc2b624564c1ec4f7d27c98e5a0074920e7ee5a1)
Branch: v4.4
https://github.com/mongodb/mongo/commit/e7b7713e9757cf07c20125c78d948663c10c6985

Comment by Githook User [ 23/Mar/22 ]

Author:

{'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}

Message: SERVER-56003 use separate run icecc scripts for multiple ninja file builds.
Branch: master
https://github.com/mongodb/mongo/commit/fc2b624564c1ec4f7d27c98e5a0074920e7ee5a1

Comment by Daniel Moody [ 17/Mar/22 ]

acm yep that'll work too, i'll update the PR

Comment by Andrew Morrow (Inactive) [ 17/Mar/22 ]

daniel.moody - The compiler package part is all handled by SCons. I wonder if we can do something silly and just tack the VARIANT_DIR onto run-icecc.sh if we are using Ninja. Basically, add a new ICECREAM_ expansion like ICECREAM_RUN_SCRIPT_SUFFIX, and then up in SConstruct , do:

if ninja:
    env['ICECREAM_RUN_SCRIPT_SUFFIX'] = '$VARIANT_DIR'

That'd get you a distinct run_icecc.sh script when using Ninja, change nothing for SCons (parts of) builds with the compiler packages still shared. We already do something similar to partition the SConf state, like CONFIGUREDIR='$BUILD_ROOT/scons/$VARIANT_DIR/sconf_temp',.

I'll admit it is a little clunky to need this extra expansion, but I think I like it a little better since it avoids any impact for SCons builds.

Comment by Daniel Moody [ 15/Mar/22 ]

acm yes this is related to two ninja files as the original description points out. Using multiple icecream directories is another solution, but that does mean duplicate compiler packages being created in other scenarios, which is somewhat insignificant, but not optimal.

Comment by Daniel Moody [ 26/Aug/21 ]

multiple ninja files does not require multiple variants (but I would not be opposed to strictly enforcing that because ninja does not support multiple ninja files in the same build, and by using variants we can technically separate the multiple ninja builds). The solution I mentioned above fixes the compiler mtime problem, which mathias mentioned, but is not the main issue in this ticket. For the run-icecc.sh, I created a per compiler package run-icecc-

{compiler-package-hash}

.sh script, agnostic to ninja or variants.

Comment by Andrew Morrow (Inactive) [ 26/Aug/21 ]

Multiple Ninja files require different variant directories, right? Could we just rescope the run-icecc.sh script to be per-variant directory?

Comment by Daniel Moody [ 24/Aug/21 ]

Proposed solution:
Instead of using mtime at all, I propose that an md5sum of the compiler's (or tools in use in general) be generated to a file. To make this cross platform, we will invoke a command line python -c hashlib one liner. It must perform the hash and update the file if it is different otherwise don't touch the file. A fake target will be used to ensure that the hash is done on every invocation of ninja, this is a drawback. Using restat, we can cause everything to rebuild only if the compiler or other tool md5 changes.

Comment by Mathias Stearn [ 12/Apr/21 ]

Just to be clear, I think that will solve a different problem, specifically the reliance on mtimes to detect compiler upgrades. The comment in that function describes it in more detail, but naive mtimes checking doesn't work because most Linux distros backdate the mtimes to well before the install time.

I don't know enough about the issue in this ticket to know if it will solve it, but I have some doubts. A possibility better solution would be to add NINJA_PREFIX to the run_icecc file name, to ensure each ninja file gets a private script. There may be other things needed to ensure it gets correctly regened if you change compilers within a single build.ninja file, but that will at least help ensure that you don't needlessly regen it when flipping between ninja files.

Generated at Thu Feb 08 05:38:02 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.