[SERVER-55129] virtualenv setup in evergreen takes too long Created: 10/Mar/21 Updated: 29/Oct/23 Resolved: 23/Aug/21 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Build |
| Affects Version/s: | None |
| Fix Version/s: | 5.1.0-rc0 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Andrew Morrow (Inactive) | Assignee: | Mikhail Shchatko |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | tig-qwin-eligible | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||
| Backport Requested: |
v5.0
|
||||||||
| Sprint: | STM 2021-08-23, STM 2021-09-06 | ||||||||
| Participants: | |||||||||
| Story Points: | 2 | ||||||||
| Description |
|
This compile task took 3:22, but of that time, the actual compile work took only 2:18: https://evergreen.mongodb.com/task_log_raw/mongodb_mongo_master_enterprise_rhel_80_64_bit_dynamic_required_compile_dist_test_6ddf0cc7c320987e801e24bf171147d1fd7a0a5b_21_03_07_11_05_12/0?type=T#L11831 We spent nearly 30 seconds preparing the virtualenv: https://evergreen.mongodb.com/task_log_raw/mongodb_mongo_master_enterprise_rhel_80_64_bit_dynamic_required_compile_dist_test_6ddf0cc7c320987e801e24bf171147d1fd7a0a5b_21_03_07_11_05_12/0?type=T#L56 That's roughly 15% of the task time(we spent about the same 15% cloning the repo). Overall, we run about 600 tasks for each build, so that's 300 minutes or 5 hours of compute time spent creating the same virtualenv. For the build above, that represents about 4% of our total compute time. We should find a way to construct these virtualenvs faster. |
| Comments |
| Comment by Vivian Ge (Inactive) [ 06/Oct/21 ] |
|
Updating the fixversion since branching activities occurred yesterday. This ticket will be in rc0 when it’s been triggered. For more active release information, please keep an eye on #server-release. Thank you! |
| Comment by Githook User [ 23/Aug/21 ] |
|
Author: {'name': 'Mikhail Shchatko', 'email': 'mikhail.shchatko@mongodb.com', 'username': 'MikhailShchatko'}Message: |
| Comment by Brooke Miller [ 15/Jul/21 ] |
|
We've punted this ticket to next sprint (ending 08/09) since we're still blocked by the linked ticket (BUILD-13222), which is currently being investigated. |
| Comment by Mikhail Shchatko [ 15/Jun/21 ] |
|
Following the approach of creating virtualenv in a separate task and using the tarball in the rest of the tasks, this is currently blocked due to macos failures (BUILD-13222): https://spruce.mongodb.com/version/60c8990061837d7e43f541b1/tasks?page=0&statuses=failed,system-failed&variant=macos |
| Comment by Brooke Miller [ 04/May/21 ] |
|
Richard explored an approach using evergreen's workdir synchronization but unfortunately there's an error when spinning up the virtualenv on windows. It may be possible to investigate that error further to potentially follow that approach. Despite that, we think it'd be best to follow the approach Rob mentioned above (installing the compile dependencies only). |