[SERVER-13287] Addition of debug symbols has doubled compile time Created: 20/Feb/14  Updated: 11/Jul/16  Resolved: 17/Apr/14

Status: Closed
Project: Core Server
Component/s: Build, Packaging
Affects Version/s: None
Fix Version/s: 2.6.1, 2.7.0

Type: Improvement Priority: Major - P3
Reporter: Ian Whalen (Inactive) Assignee: Matt Kangas
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Backport Completed:
Participants:

 Description   

30 minutes:
http://mci.10gen.com/ui/task/mongodb_mongo_master_linux_64_05c1dd31bbb2f5e039845031213f1b1f243d598f_14_02_19_18_00_06_compile
http://mci.10gen.com/ui/task_log_raw/mongodb_mongo_master_linux_64_05c1dd31bbb2f5e039845031213f1b1f243d598f_14_02_19_18_00_06_compile?type=T

12 minutes:
http://mci.10gen.com/ui/task/mongodb_mongo_master_linux_64_dd56209bfcb34d59191d1b3a9caa1af177f8e16f_14_02_19_15_44_06_compile
http://mci.10gen.com/ui/task_log_raw/mongodb_mongo_master_linux_64_dd56209bfcb34d59191d1b3a9caa1af177f8e16f_14_02_19_15_44_06_compile?type=T

Is this proper/expected? Or could this be some kind of misconfiguration?

cc acm to add input if needed.



 Comments   
Comment by Githook User [ 17/Apr/14 ]

Author:

{u'username': u'deafgoat', u'name': u'Wisdom Omuya', u'email': u'deafgoat@gmail.com'}

Message: SERVER-13287 delegate to shell when compressing tar archives

Signed-off-by: Matt Kangas <matt.kangas@mongodb.com>
(cherry picked from commit b34245c1ff87833a307b2eac76bb710ec5033d28)
Branch: v2.6
https://github.com/mongodb/mongo/commit/a093bfa703c4a63fb26b1bc16c0f6ae13261486e

Comment by Adinoyi Omuya [ 17/Apr/14 ]

You're right! I completely missed it as well, sorry about the mistake.

Looks like compile times are significantly lower across all linux/unix flavor buildvariants.

Comment by Matt Kangas [ 17/Apr/14 ]

adinoyi.omuya@10gen.com - your commits in pull/658 said "MCI-13287". Should have read "SERVER-13287".

I squashed them and tested locally, but I failed to notice this before pushing to master.

commit b34245c1ff87833a307b2eac76bb710ec5033d28
Author:     Wisdom Omuya <deafgoat@gmail.com>
AuthorDate: Wed Mar 19 18:20:06 2014 -0400
Commit:     Matt Kangas <matt.kangas@mongodb.com>
CommitDate: Wed Apr 16 23:48:32 2014 -0400
 
    MCI-13287: delegate to shell when compressing tar archives
 
    Signed-off-by: Matt Kangas <matt.kangas@mongodb.com>

Comment by Githook User [ 26/Mar/14 ]

Author:

{u'username': u'monkey101', u'name': u'Dan Pasette', u'email': u'dan@10gen.com'}

Message: Revert "SERVER-13287: delegate to shell when compressing tar archives"

This reverts commit e73399e0e696aaaf1e859533e10cc2b74d46994f.
Branch: master
https://github.com/mongodb/mongo/commit/2d1f1b7c5e6a42a503dacee442ec8ebaa11a9ead

Comment by Githook User [ 25/Mar/14 ]

Author:

{u'username': u'deafgoat', u'name': u'Wisdom Omuya', u'email': u'deafgoat@gmail.com'}

Message: SERVER-13287: delegate to shell when compressing tar archives

Signed-off-by: Matt Kangas <matt.kangas@mongodb.com>
Branch: master
https://github.com/mongodb/mongo/commit/e73399e0e696aaaf1e859533e10cc2b74d46994f

Comment by Andrew Morrow (Inactive) [ 20/Feb/14 ]

Yeah make_archive is a bit of a travesty. SCons knows how to make archives by way of its Package rule, but we don't use that. I see no reason that making make_archive slightly worse (one more level of shelling) to make build times better (we all rejoice), would be a bad tradeoff.

Comment by Michael O'Brien [ 20/Feb/14 ]

For example, see the log timing here:
http://mci.10gen.com/ui/task_log_raw/mongodb_mongo_master_linux_64_05c1dd31bbb2f5e039845031213f1b1f243d598f_14_02_19_18_00_06_compile?type=T#L3077

Suspicious code in make_archive.py:
https://github.com/mongodb/mongo/blob/master/buildscripts/make_archive.py#L85

Comment by Michael O'Brien [ 20/Feb/14 ]

The issue is in mongo/buildscripts/make_archive.py which is called by scons to generate the archive with nonstripped binaries.
Apparently builds the archive via python libs (tarfile and zipfile) which is super slow.
We're looking to see if we can replace it with shell calls to tar zcf instead, which should be much faster.

Comment by Andrew Morrow (Inactive) [ 20/Feb/14 ]

ian@10gen.com Is there a link to the MCI config change? The listed git hashes for the mongodb repo don't have anything to do with debug symbols.

Generated at Thu Feb 08 03:31:14 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.