[SERVER-49036] the compile_commands.json produced by ninja is incomplete. Created: 23/Jun/20  Updated: 29/Oct/23  Resolved: 07/Jul/20

Status: Closed
Project: Core Server
Component/s: Build
Affects Version/s: None
Fix Version/s: 4.4.1, 4.7.0

Type: Bug Priority: Major - P3
Reporter: Gabriel Russell (Inactive) Assignee: Daniel Moody
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.4
Sprint: Dev Platform 2020-07-13
Participants:

 Description   

The compile_commands.json produced by scons is complete.

For example, this is not enough information to compile this target:

  {
    "directory": "/home/gabriel/git/mongo/master",
    "command": "PATH=/opt/mongodbtoolchain/v3/bin:/usr/local/bin:/opt/bin:/bin:/usr/bin /opt/mongodbtoolchain/v3/bin/clang @build/ninja/third_party/zstandard-1.4.4/zstd/lib/dictBuilder/zdict.dyn.o.rsp",
    "file": "src/third_party/zstandard-1.4.4/zstd/lib/dictBuilder/zdict.c",
    "output": "build/ninja/third_party/zstandard-1.4.4/zstd/lib/dictBuilder/zdict.dyn.o"
  }



 Comments   
Comment by Githook User [ 13/Oct/20 ]

Author:

{'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}

Message: SERVER-50363 Merge --build-tools=next into stable

Merging the following fixes into the stable version of the build tools
and migrate fully from --ninja=next to the --build-tools=next:

(cherry picked from commit 78bb3f3c8a658a5a9fec8d55864e426382f68bd0)
Branch: v4.4
https://github.com/mongodb/mongo/commit/e7f6696a96db5438a269c5e67a06ff9e05cb897a

Comment by Githook User [ 07/Oct/20 ]

Author:

{'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}

Message: SERVER-50363 Merge --build-tools=next into stable

Merging the following fixes into the stable version of the build tools
and migrate fully from --ninja=next to the --build-tools=next:

(cherry picked from commit 78bb3f3c8a658a5a9fec8d55864e426382f68bd0)
Branch: v4.4
https://github.com/mongodb/mongo/commit/7b93770e0835087c023ae3e3f36eeb46d204767f

Comment by Githook User [ 06/Oct/20 ]

Author:

{'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}

Message: SERVER-50363 Merge --build-tools=next into stable

Merging the following fixes into the stable version of the build tools
and migrate fully from --ninja=next to the --build-tools=next:

Comment by Githook User [ 06/Aug/20 ]

Author:

{'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}

Message: SERVER-49036 expand response files for generating the compdb from ninja

(cherry picked from commit 5fe923a0aa312044062df044eb4eaa47951f70ec)
Branch: v4.4
https://github.com/mongodb/mongo/commit/fa7f6842d53431ef679a587ba20b62cbe49d871c

Comment by Andrew Morrow (Inactive) [ 02/Jul/20 ]

daniel.moody - We will want a backport to v4.4 for this.

Comment by Githook User [ 02/Jul/20 ]

Author:

{'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}

Message: SERVER-49036 expand response files for generating the compdb from ninja
Branch: master
https://github.com/mongodb/mongo/commit/5fe923a0aa312044062df044eb4eaa47951f70ec

Comment by Daniel Moody [ 29/Jun/20 ]

Alternatively, if we like the compilation database generated by SCons better, should we reconsider having Ninja generate at all, and have the Ninja `compiledb` simply re-invoke SCons?

 

I am curious how well that would work. Assuming we force compilation db to the last build step, all nodes could be built by ninja, and scons would finish up with a compilation db. I would wonder if they would generate compilation databases that we would consider equivalent? I would assume it would be slower, but not terribly slower, and probably not be reinvoked for iterative rebuilds. If the investigation pans out I think that would be the best path forward.

Comment by Daniel Moody [ 29/Jun/20 ]

What released versions of Ninja contain that flag? 

 

Looking at the tags in this commit: https://github.com/ninja-build/ninja/commit/26341ca0d652b6e50aa57fa176bf610480be5edd
looks like it was introduced in 1.9

 

Should we instead be telling people to use keeprsp as suggested in the Ninja PR?

 

The downside to that is that it will keep all response files hanging around. Its not terrible, but there are other bugs I have seen in older ninja (like dependency bugs), so I would prefer to force new ninjas if possible. If we roll it all up with python, then we can just add it to mongo requirements.txt for the build.

Comment by Daniel Moody [ 29/Jun/20 ]

As of right now, we don't require use of any particular version of Ninja.

 

One tangental note, I would recommend making the build use ninja installed with pip. That makes it dependent on python which the build system is using, and you can use python module system to find the binary and the ninja_syntax.py file reliably. Then you don't need to worry about what version the system package manager is installing or other cross platform issues.

Comment by Andrew Morrow (Inactive) [ 29/Jun/20 ]

What released versions of Ninja contain that flag? As of right now, we don't require use of any particular version of Ninja. Should we instead be telling people to use keeprsp as suggested in the Ninja PR?

Alternatively, if we like the compilation database generated by SCons better, should we reconsider having Ninja generate at all, and have the Ninja `compiledb` simply re-invoke SCons? Our SCons level compilation_database actually works a little bit better in some ways: it has a dependency on the generated sources so ensures their existence.

Comment by Daniel Moody [ 29/Jun/20 ]

ninja supports response file expansion with the -x option, although its not in their documentation directly: https://github.com/ninja-build/ninja/pull/1223

 

Also adding the -x option will still be an incomplete compilation database, because the substfiles and textfiles that the build uses can only be done in scons, and scons must be reinvoked to generate those.

 

One option would be to modifiy these scons only targets and write external scripts which ninja can use to generate them. This would allow the compdb that ninja spits out to represent the full build. (i.e. there would be no build using the TEMPLATE rule in the generated ninja file)

Generated at Thu Feb 08 05:18:46 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.