[SERVER-51754] Running `ninja compiledb` generates much smaller file than `scons compiledb` Created: 20/Oct/20  Updated: 27/Oct/23  Resolved: 26/Oct/20

Status: Closed
Project: Core Server
Component/s: Build
Affects Version/s: 4.8.1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Kaloian Manassiev Assignee: [DO NOT ASSIGN] Backlog - Server Development Platform Team (SDP) (Inactive)
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Assigned Teams:
Server Development Platform
Operating System: ALL
Participants:

 Description   

I use VisualStudio Code often and because of this I rely on compile_commands.json.

When I run ninja compiledb, I get a file of 1.8M:

[kaloianm@Kaloian's-MacBook-Pro-(Work):~/workspace/mongo]$ ninja compiledb
[0/1] Building compile_commands.json
[kaloianm@Kaloian's-MacBook-Pro-(Work):~/workspace/mongo]$ ls -alh compile_commands.json
-rw-r--r--  1 kaloianm  staff   1.8M Oct 20 08:15 compile_commands.json

When I run scons compiledb, I get a file of 8.4M:

[kaloianm@Kaloian's-MacBook-Pro-(Work):~/workspace/mongo]$ buildscripts/scons.py --ssl --variables-files=etc/scons/xcode_macosx.vars --libc++ --detect-odr-violations --dbg=on --opt=off --build-tools=next VARIANT_DIR=ninja compiledb
scons: Reading SConscript files ...
...
scons: done reading SConscript files.
scons: Building targets ...
Building compilation database compile_commands.json
scons: done building targets.
[kaloianm@Kaloian's-MacBook-Pro-(Work):~/workspace/mongo]$ ls -alh compile_commands.json
-rw-r--r--  1 kaloianm  staff   8.4M Oct 20 08:19 compile_commands.json



 Comments   
Comment by Daniel Moody [ 26/Oct/20 ]

Kal confirmed that updating to a newer version of ninja fixed the issue

Comment by Daniel Moody [ 23/Oct/20 ]

Ninja should be expanding the response files which was fixed in this commit: https://github.com/mongodb/mongo/commit/5fe923a0aa312044062df044eb4eaa47951f70ec 

So I expect it to be working for you.

Comment by Kaloian Manassiev [ 23/Oct/20 ]

Sorry for the late reply, Daniel. You are correct – the one generated by scons indeed expands the paths:

        "command": "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ -o build/ninja/mongo/db/exec/document_value/document.o -c -Woverloaded-virtual -Werror=unused-result -Wpessimizing-move -Wredundant-move -Wno-undefined-var-template -Wno-instantiation-after-specialization -fsized-deallocation -Wno-defaulted-function-deleted -Wunused-exception-parameter -stdlib=libc++ -std=c++17 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -mmacosx-version-min=10.13 -target darwin17.0.0 -arch x86_64 -fno-omit-frame-pointer -fno-strict-aliasing -fasynchronous-unwind-tables -ggdb -pthread -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -Werror -O0 -Wno-unused-local-typedefs -Wno-unused-function -Wno-unused-private-field -Wno-deprecated-declarations -Wno-tautological-constant-out-of-range-compare -Wno-tautological-constant-compare -Wno-tautological-unsigned-zero-compare -Wno-tautological-unsigned-enum-zero-compare -Wno-unused-const-variable -Wno-missing-braces -Wno-inconsistent-missing-override -Wno-potentially-evaluated-expression -Wno-unused-lambda-capture -Wno-exceptions -Wunguarded-availability -fstack-protector-strong -fno-builtin-memcmp -DSAFEINT_USE_INTRINSICS=0 -DPCRE_STATIC -DBOOST_THREAD_VERSION=5 -DBOOST_THREAD_USES_DATETIME -DBOOST_SYSTEM_NO_DEPRECATED -DBOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS -DBOOST_ENABLE_ASSERT_DEBUG_HANDLER -DBOOST_LOG_NO_SHORTHAND_NAMES -DBOOST_LOG_USE_NATIVE_SYSLOG -DBOOST_LOG_WITHOUT_THREAD_ATTR -DABSL_FORCE_ALIGNED_ACCESS -Isrc/third_party/s2 -Isrc/third_party/variant-1.4.0/include -Isrc/third_party/SafeInt -Isrc/third_party/pcre-8.42 -Isrc/third_party/fmt/dist/include -Isrc/third_party/boost-1.70.0 -Isrc/third_party/abseil-cpp-master/abseil-cpp -Ibuild/ninja -Isrc src/mongo/db/exec/document_value/document.cpp",
        "directory": "/Users/kaloianm/workspace/mongo",
        "file": "src/mongo/db/exec/document_value/document.cpp"

and the one from ninja doesn't:

    "directory": "/Users/kaloianm/workspace/mongo",
    "command": "export PATH='/usr/local/bin:/opt/bin:/bin:/usr/bin';export PATHOSX='/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin';/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ @build/ninja/mongo/bson/mutable/document.o.rsp",
    "file": "src/mongo/bson/mutable/document.cpp"

Feel free to close this ticket then.

Comment by Daniel Moody [ 20/Oct/20 ]

Testing on ubuntu18 workstation, I get ninja compledb at about 8.2 MB and the scons compiledb at 7.5 MB. The builds are not in absolute parity, and there are few options ninja adds to each compile command so I would expect ninja's compiledb to be a little bigger.

In your case I would guess that ninja did not expand the response files, so if you look in your compile commands json, you should se the @filepath.rsp instead of the command line options.

This ticket added support for expanding the ninja response files in the compiledb target, but only certain versions of ninja support that option. The build should do a configure check to see if it can use that option, so you should see it in the configure output. Which version of ninja are you running on your laptop? I tested on my macbook and which has ninja 1.10 installed and it generated a compiledb almost the same size as on the linux workstation.

Generated at Thu Feb 08 05:26:20 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.