[SERVER-50372] ninja and SCons do not see each others ccache Created: 18/Aug/20  Updated: 03/Sep/20  Resolved: 03/Sep/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Daniel Moody Assignee: Ryan Egesdahl (Inactive)
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File ninja_ccache.log     Text File scons_ccache.log    
Operating System: ALL
Steps To Reproduce:

reproduced on master commit: 0662433ac038f4f3cf137a1be30ca245c180e990

Repro system:
Ubuntu 18.04 virtual workstation

Steps to reproduce:

  1. Start with clean checkout and clean cache:
    git clean -xfd
    ccache -Cz
  2. build with scons
    /usr/bin/python3.8 "buildscripts/scons.py" CCACHE=ccache build/opt/mongo/db/global_settings.o
  3. delete the object file and set ccache to log:
    rm build/opt/mongo/db/global_settings.o
    export CCACHE_LOGFILE=scons_ccache.log
  4. build again
    /usr/bin/python3.8 "buildscripts/scons.py" CCACHE=ccache build/opt/mongo/db/global_settings.o
  5. now set the log file to something else:
    export CCACHE_LOGFILE=ninja_ccache.log
  6. generate ninja and build
    /usr/bin/python3.8 "buildscripts/scons.py" CCACHE=ccache --ninja --build-tools=next generate-ninja
    ninja build/opt/mongo/db/global_settings.o
  7. Now remove the file and cache log so we can minimize the log
    rm ninja_ccache.log
    rm build/opt/mongo/db/global_settings.o
    ninja build/opt/mongo/db/global_settings.o
  8. compare the log files.
Sprint: Dev Platform 2020-09-07
Participants:

 Description   

When building first with SCons or ninja, then deleting the built object file, and then rebuilding with the opposing tool, SCons and ninja do not use each others ccache and instead will build and put the object file in ccache twice with different hashes.

 

Attached log files show two cache hits, but with different hashes.



 Comments   
Comment by Ryan Egesdahl (Inactive) [ 03/Sep/20 ]

The reason this happens is because Ninja adds some arguments to the compiler command line, such as -MMD, that SCons does not and does not need. In particular, these options are present to allow Ninja to do dependency tracking on files, which SCons does internally (and, frankly, in a much more reliable way).

While we could possibly do some work trying to add the options to the compiler command line produced by SCons, the argument order would be difficult to get right, and this matters because arugment order affects the hash produced by ccache. And as stated previously, SCons does not actually need these options to do dependency tracking.

Because of the amount of work likely to be involved in making the hashes match combined with the fact that the work would be superfluous with SCons and likely pointless given how developers generally don't switch between pure SCons and Ninja builds, we feel that this effort would be unprofitable. We would like to instead focus our efforts on making SCons incremental builds fast enough that developers don't need Ninja anymore.

Comment by April Schoffer [ 03/Sep/20 ]

To write up rationale for won't fix.

Generated at Thu Feb 08 05:22:29 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.