[SERVER-38389] ccache: error: execv of @/tmp/tmpCwyvtv.lnk failed: No such file or directory Created: 04/Dec/18  Updated: 07/Dec/18  Resolved: 05/Dec/18

Status: Closed
Project: Core Server
Component/s: Build
Affects Version/s: 4.0.4
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Connecting Media Assignee: Andrew Morrow (Inactive)
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File build.log    
Operating System: ALL
Steps To Reproduce:
  • Try to build version 4.0.4 with the command { {buildscripts/scons.py AR="aarch64-linux-gnu-ar" CC="ccache aarch64-linux-gnu-gcc" CXX="ccache aarch64-linux-gnu-g++" HOST_ARCH="x86_64" TARGET_ARCH="aarch64" CCFLAGS="-march=armv8-a+crc" --disable-warnings-as-errors -j 6 mongod}

    }

Sprint: Dev Tools 2018-12-17
Participants:

 Description   

When trying to crosscompile MongoDB 4.0.4 for arm64/aarch64, the build fails with this rather strange error message:

[2018-12-04 15:48:56 (01:41:57.323047)] ccache @/tmp/tmpCwyvtv.lnk
[2018-12-04 15:48:56 (01:41:57.544064)] ccache: error: execv of @/tmp/tmpCwyvtv.lnk failed: No such file or directory

The command line used for the build was buildscripts/scons.py AR="aarch64-linux-gnu-ar" CC="ccache aarch64-linux-gnu-gcc" CXX="ccache aarch64-linux-gnu-g++" HOST_ARCH="x86_64" TARGET_ARCH="aarch64" CCFLAGS="-march=armv8-a+crc" --disable-warnings-as-errors -j 6 mongod

The complete build log is attached as a file.
Note that the timestamping is happening through my build system. When running from a normal command line it's not there.



 Comments   
Comment by Connecting Media [ 07/Dec/18 ]

Awesome.

One thing I'd like to point out is that I am using ccache because it has already been set up and is properly configured for the entire build system. And I'm building a fair bit more than just MongoDB. So I'd like to keep all caches in one place.

Comment by Andrew Morrow (Inactive) [ 06/Dec/18 ]

I've started a thread on scons-users to discuss: https://pairlist4.pair.net/pipermail/scons-users/2018-December/007468.html

Comment by Andrew Morrow (Inactive) [ 06/Dec/18 ]

ConnectingMedia - Yes, thanks.

Comment by Connecting Media [ 06/Dec/18 ]

And in case you haven't seen it I contacted you via email. I put the link in the title and my mail's domain is connectingmedia.de.

Comment by Andrew Morrow (Inactive) [ 06/Dec/18 ]

Yes, probably. I will raise this ticket with one of the SCons developers that I know.

Comment by Connecting Media [ 06/Dec/18 ]

Ok. Thanks for your time. Do you think that issue should be reported to SCons?

Comment by Andrew Morrow (Inactive) [ 05/Dec/18 ]

I'm closing this as Won't Fix, because it appears to be actually a bug or limitation in SCons, and there are several viable workarounds.

Comment by Andrew Morrow (Inactive) [ 05/Dec/18 ]

Having taken a closer look, I'm fairly sure this is a bug in the SCons implementation of temp files for long command lines, where it assumes that the first token of the command line is the the thing that should interpret the file. That doesn't seem to be correct when using ccache. On the other hand, it sort of feels like ccache would need to do so, because otherwise how can it know that the compilation is a "match" to a previous invocation, unless it examines the arguments passed to the compiler, which are in the file.

Overall, I'd suggest just avoiding this situation, by either adjusting the MAXLINELENGTH as you have done, or by using the "symlink" method for ccache, or by abandoning ccache and using the built-in SCons caching.

Comment by Andrew Morrow (Inactive) [ 05/Dec/18 ]

ConnectingMedia - Good observation, I'll take another look. Perhaps there is a bug in SCons where it is assuming that only the first argument of the "compiler" should be propagated to the response file investigation (or in our setup for using response files).

Regarding your other notes:

  • Yes, using `getconf` is a fine way. Anything that works for you really.
  • Regarding your question about installation, I think it really depends on whether /path/to/install is writable by the user running SCons to do the overall build. If so, sure, the one shot build should just work. If not, you probably want to do two invocations, so that you don't run the whole compile step under sudo. The overall "installation" setup for our builds is something we are looking to improve, so behavior here is likely to change in the future.
  • It is normal that the binaries are that large, but the majority of it is debugging information. If you don't need that, you can strip the binaries, or manually use objcopy to strip the debug info to another file and add the necessary .gnu_debuglink section.

In the future, if you have build system questions, you can reach out on mongodb-dev first and we can discuss. It is moderated so your posts won't be immediately visible. I'm also happy to see that you were able to figure out how to get an aarch64 cross of MongoDB going!

Finally, would you mind sharing some information on your plans regarding MongoDB on ARM? It is an area of personal interest for me. Feel free to reach out via email (it is in my JIRA profile).

I'm going to leave this ticket open while I take another look at the temp file handling when the "compiler" is two arguments, but given that you have a workaround, I may decide not to fix it.

Comment by Connecting Media [ 05/Dec/18 ]

And is it normal the mongod binary is 1GiB in size? (mongo and mongos are several hundred MiB too)

Comment by Connecting Media [ 05/Dec/18 ]

From the looks of it, adding MAXLINELENGTH="$(getconf ARG_MAX)" solved the issue.

Comment by Connecting Media [ 05/Dec/18 ]

Interesting. From what I saw in my build it called aarch64-linux-gnu-g++ @/tmp/tmp3QOOvI.lnk. That worked fine. However in the build where it fails it's just ccache @/tmp/tmpCwyvtv.lnk instead of the correct compiler call which should be ccache aarch64-linux-gnu-g++ @/tmp/tmpCwyvtv.lnk. So to it looks like that the script isn't using the right call at the end.
And I think MAXLINELENGTH="$(getconf ARG_MAX)" should work nicely, so I don't have to hardcode a random value but instead can use the actual systems limit.

Another slightly unrelated question (I haven't really found a good place to ask about build stuff anywhere besides this place. So my apologies if it's inappropriate to ask that here) is if it's possible to call buildscripts/scons.py <parameters> --prefix /path/to/install mongod install or if they need to be two different calls.

Comment by Andrew Morrow (Inactive) [ 04/Dec/18 ]

OK, I can reproduce this. It appears that ccache simply doesn't understand response files, which is a little weird. You can work around this by adding
MAXLINELENGTH=<some-very-big-number> to your SCons invocation, and it won't try to use them. If that works, great. But if you do run into the maximum command line length on your local system, then it won't. At that point, you would just need to not use ccache. As I mentioned, SCons has its own cache, so you can use that too.

Comment by Andrew Morrow (Inactive) [ 04/Dec/18 ]

Sounds good. Meanwhile I'll see if I can reproduce your problem when using ccache or not.

Comment by Connecting Media [ 04/Dec/18 ]

I will try it without ccache. Will get back to you tomorrow, as I'll leave work before the build will finish (or fail).

Comment by Andrew Morrow (Inactive) [ 04/Dec/18 ]

Does the build work if you stop using ccache? Note that SCons has its own caching subsystem, which you can enable with --cache. You can control where it caches with --cache-dir.

Generated at Thu Feb 08 04:48:49 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.