[SERVER-37507] Reducing disk space needed to build mongodb Created: 07/Oct/18  Updated: 27/Oct/23  Resolved: 15/Oct/18

Status: Closed
Project: Core Server
Component/s: Build
Affects Version/s: 3.6.8
Fix Version/s: None

Type: Question Priority: Minor - P4
Reporter: Ryan Schmidt Assignee: Andrew Morrow (Inactive)
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Sprint: Dev Tools 2018-10-22
Participants:

 Description   

Building mongodb 3.6.8 takes about 15GB of disk space on macOS. This has been a problem for the MacPorts project's build farm, which I administer, because our build machines don't always have that much disk space available.

It's hard to imagine what all that disk space is being used for, since the total size of everything that actually ends up getting installed is under 200MB. I have not yet had an opportunity to try building mongodb 4, but assuming the build system hasn't changed that much, is there a way to reduce how much disk space it uses to build? Or are there any flags to tell the build system to clean up intermediate files as it goes along?



 Comments   
Comment by Ryan Schmidt [ 15/Oct/18 ]

Thanks for your help. I've updated mongodb in MacPorts to 4.0.3 leaving the -ggdb flag untouched.

Comment by Andrew Morrow (Inactive) [ 15/Oct/18 ]

ryandesign - I'm going to close this as "Works as Designed". If you need further help with this please feel free to re-open or comment on the ticket and I will see it.

Comment by Andrew Morrow (Inactive) [ 10/Oct/18 ]

ryandesign - Unless you are running dsymutil on the resulting mongod, mongos binaries and retaining the produced .dSYM directories somewhere, the binaries that you are currently installing are effectively already stripped of debug info once you throw away the object files. I can't entirely explain what the 40MB savings in the final binaries is about when you entirely disable debug information, but maybe it is whatever minimal debug information is needed to allow reaching out to any .dSYM files? I have a hard time believing that that would be 40 MB, but who knows. It is worth exploring. In any event, if you aren't making those .dSYM files, you aren't doing any worse by disabling debug info entirely, I think. That said, my recommendation would be not to change it, if you can live with 5.6GB for the build.

Comment by Ryan Schmidt [ 10/Oct/18 ]

You mean -ggdb? I tried removing that and it does reduce the disk space needed down to 900MB. The installed files make up 156MB of that, so they've decreased in size too; presumably they no longer contain debug information. Not sure how important the debug information would be to a user, or to you—would this negatively affect the readability of crash reports that a user might send you, for example?

I also tried replacing -ggdb with -g, which didn't change disk usage much (5.6GB).

Comment by Andrew Morrow (Inactive) [ 09/Oct/18 ]

ryandesign - Great news, I'm glad that helped. FWIW, I think the reason you still see 5.9 GB of disk space despite the final binary only being 191MB is that OS X ld leaves debug info in the object files and doesn't copy it into the binaries. It still feels pretty high to me though, so if you wanted to test that theory, you could edit the -g out of CCFLAGS in the top level SConstruct file and see what your disk utilization looks like. I'm not sure what MacPorts approach to handling separate debug info looks like, but again, if you aren't running dsymutil and attaching debug packages, you probably don't need the debug info in the object files in the first place. We don't currently offer a build flag to do that since we more or less always want the debug info, but it wouldn't be too hard to add.

Comment by Ryan Schmidt [ 09/Oct/18 ]

Thanks for the suggestion! You're right, we haven't configured our mongodb port to run any tests. Changing the build targets from all mongobridge to core mongobridge seems to produce the same files to install, but only uses 5.9GB of disk space to build. That will help us a lot. 5.9GB is still a lot of space to install just 191MB, but it's much better than 15GB.

Comment by Andrew Morrow (Inactive) [ 09/Oct/18 ]

ryandesign - I see that you are building the all target, which includes all of the unit tests, all of which are statically linked. It also doesn't appear to me that you are actually running those tests. I think we can probably greatly reduce the overhead by being more selective about the targets to build.

Comment by Ryan Schmidt [ 08/Oct/18 ]

Our mongodb Portfile is here. What it ends up running is:

scons -j8 all mongobridge CC="/usr/bin/clang" CCFLAGS="-arch x86_64" CPPPATH="/opt/local/include" CXX="/usr/bin/clang++" LIBPATH="/opt/local/lib" LINKFLAGS="-arch x86_64" TARGET_ARCH=x86_64 --disable-warnings-as-errors --libc++ --ssl --wiredtiger=on VARIANT_DIR=MP

Comment by Andrew Morrow (Inactive) [ 08/Oct/18 ]

ryandesign - Can you point me to the MacPorts script that invokes SCons? I'd like to know what targets you are building to make sure that you aren't building more than you need. I don't believe there are any options to do incremental cleanups.

Generated at Thu Feb 08 04:46:12 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.