[SERVER-55129] virtualenv setup in evergreen takes too long Created: 10/Mar/21  Updated: 29/Oct/23  Resolved: 23/Aug/21

Status: Closed
Project: Core Server
Component/s: Build
Affects Version/s: None
Fix Version/s: 5.1.0-rc0

Type: Improvement Priority: Major - P3
Reporter: Andrew Morrow (Inactive) Assignee: Mikhail Shchatko
Resolution: Fixed Votes: 0
Labels: tig-qwin-eligible
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Backwards Compatibility: Fully Compatible
Backport Requested:
v5.0
Sprint: STM 2021-08-23, STM 2021-09-06
Participants:
Story Points: 2

 Description   

This compile task took 3:22, but of that time, the actual compile work took only 2:18: https://evergreen.mongodb.com/task_log_raw/mongodb_mongo_master_enterprise_rhel_80_64_bit_dynamic_required_compile_dist_test_6ddf0cc7c320987e801e24bf171147d1fd7a0a5b_21_03_07_11_05_12/0?type=T#L11831

We spent nearly 30 seconds preparing the virtualenv: https://evergreen.mongodb.com/task_log_raw/mongodb_mongo_master_enterprise_rhel_80_64_bit_dynamic_required_compile_dist_test_6ddf0cc7c320987e801e24bf171147d1fd7a0a5b_21_03_07_11_05_12/0?type=T#L56

That's roughly 15% of the task time(we spent about the same 15% cloning the repo).

Overall, we run about 600 tasks for each build, so that's 300 minutes or 5 hours of compute time spent creating the same virtualenv. For the build above, that represents about 4% of our total compute time.

We should find a way to construct these virtualenvs faster.



 Comments   
Comment by Vivian Ge (Inactive) [ 06/Oct/21 ]

Updating the fixversion since branching activities occurred yesterday. This ticket will be in rc0 when it’s been triggered. For more active release information, please keep an eye on #server-release. Thank you!

Comment by Githook User [ 23/Aug/21 ]

Author:

{'name': 'Mikhail Shchatko', 'email': 'mikhail.shchatko@mongodb.com', 'username': 'MikhailShchatko'}

Message: SERVER-55129 Archive python venv in archive_dist_test and fetch it
Branch: master
https://github.com/mongodb/mongo/commit/9fbd37083621ef313636552da0045a2d2a2d8955

Comment by Brooke Miller [ 15/Jul/21 ]

We've punted this ticket to next sprint (ending 08/09) since we're still blocked by the linked ticket (BUILD-13222), which is currently being investigated.

Comment by Mikhail Shchatko [ 15/Jun/21 ]

Following the approach of creating virtualenv in a separate task and using the tarball in the rest of the tasks, this is currently blocked due to macos failures (BUILD-13222): https://spruce.mongodb.com/version/60c8990061837d7e43f541b1/tasks?page=0&statuses=failed,system-failed&variant=macos

Comment by Brooke Miller [ 04/May/21 ]

Richard explored an approach using evergreen's workdir synchronization but unfortunately there's an error when spinning up the virtualenv on windows. It may be possible to investigate that error further to potentially follow that approach. Despite that, we think it'd be best to follow the approach Rob mentioned above (installing the compile dependencies only).

Generated at Thu Feb 08 05:35:33 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.