[SERVER-37428] Sys-perf: linux builds using enterprise bits Created: 02/Oct/18  Updated: 29/Oct/23  Resolved: 26/Nov/18

Status: Closed
Project: Core Server
Component/s: Performance
Affects Version/s: None
Fix Version/s: 4.0.6, 4.1.6

Type: Bug Priority: Major - P3
Reporter: David Daly Assignee: Max Hirschhorn
Resolution: Fixed Votes: 0
Labels: tig-evgconfig
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Related
related to SERVER-38305 Move all sys-perf compile tasks over ... Closed
is related to SERVER-37438 System Performance: Compile Enterpris... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.0, v3.6, v3.4
Sprint: STM 2018-12-03
Participants:
Story Points: 2

 Description   

Note: this only affects internal testing systems. 

The system_perf.yml file support for enterprise modules is doing the wrong thing for non-enterprise builds. There are three compile tasks in one variant, but they all get the enterprise module. The fix seems to be making the enterprise build more like the wtdevelop build, by checking out the enterprise module to a directory outside the mongo source directory, and actively copying it in for the enterprise build as is done in "use wiredtiger develop" function. 



 Comments   
Comment by Githook User [ 21/Dec/18 ]

Author:

{'username': 'visemet', 'email': 'max.hirschhorn@mongodb.com', 'name': 'Max Hirschhorn'}

Message: SERVER-37428 Move compile_* tasks to separate build variants.

Build variants are able to define separate lists of modules whereas
tasks cannot.

(cherry picked from commit d11e4a25014c2d6a4939481dbefa52fd11d8f286)
Branch: v4.0
https://github.com/mongodb/mongo/commit/3dabdbc554a857b092af272ed3d17b55292ee6b5

Comment by Githook User [ 26/Nov/18 ]

Author:

{'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}

Message: SERVER-37428 Move compile_* tasks to separate build variants.

Build variants are able to define separate lists of modules whereas
tasks cannot.
Branch: master
https://github.com/mongodb/mongo/commit/d11e4a25014c2d6a4939481dbefa52fd11d8f286

Comment by Henrik Ingo (Inactive) [ 19/Nov/18 ]

Note that after EVG-727 / EVG-5379 this ticket changes in nature: We could now have a single compile task and use the enterprise version for everything. (Will still need a separate wtdevelop compile, of course.)

Comment by Henrik Ingo (Inactive) [ 05/Oct/18 ]

Already commented on CR, only now seeing this thread...

Is there a reason (e.g. how task dependencies work in Evergreen) that we chose to have 1 build variant with 3 compile* tasks rather than 3 build variants with 1 compile task each?

This is a good point and IMO your proposal is also an acceptable solution to the same proposal.

The reason the compile variant is the same is that we want all binaries to be compiled on the same linux and same toolchain to maintain ability to compare them. (More generally, until now we never targeted different OSes, compilers or libraries in sys-perf, focus was solely on MongoDB's own performance in an otherwise static environment.) So it was just natural to have a single variant for that.

I tried to think about why I instinctively prefer the wtdevelop approach to the one seen in enterprise module, and came to this conclusion: In the wtdevelop compile, it's the same block of code that takes one parameter and:

  • actively includes wtdevelop code in the src/ tree
  • adds -wtdevelop into the version string
  • add -wtdevelop into the filename of the tar archive

For enterprise, it is now a completely separate input parameter/mechanism that clones enterprise source code into src/ and then a separate block of code to add -enterprise to the binary or not. When I look at the code, it seems just a matter of time before someone will again accidentally include enterprise module into the community compile, and there's nothing in the subsequent code that will fail hard or alert the user about this.

If we wanted to go with your proposal, then we IMO need to change this code so that it doesn't take a parameter rather correctly determines whether the compile is enterprise or not.

What should happen in the accidental use case is that the compile task uploads ...-enterprise.tgz and the task will try to download ...tgz and fail when it isn't there. (And also mongo --version should print the truth.)

Comment by Max Hirschhorn [ 05/Oct/18 ]

There's another option of leaving the git.get_project command and build variants as-is and instead using the --modules command line option to SCons to get it to ignore the src/mongo/db/modules/enterprise/ directory in the compile and compile_wtdevelop tasks.

Comment by Max Hirschhorn [ 02/Oct/18 ]

The fix seems to be making the enterprise build more like the wtdevelop build, by checking out the enterprise module to a directory outside the mongo source directory, and actively copying it in for the enterprise build as is done in "use wiredtiger develop" function.

I don't agree with this proposal. The WiredTiger develop does what it does because there's an existing copy of the wiredtiger/wiredtiger repository vendored in the mongodb/mongo source tree - that isn't the case for the enterprise module. We shouldn't be specifying enterprise in the modules list for the build variant when we specifically are trying not to build with the enterprise module enabled.

Is there a reason (e.g. how task dependencies work in Evergreen) that we chose to have 1 build variant with 3 compile* tasks rather than 3 build variants with 1 compile task each?

Generated at Thu Feb 08 04:45:58 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.