[SERVER-70839] Spawning dynamically linked mongod processes takes ~30 secs on EVG Created: 01/Aug/22 Updated: 14/Dec/22 Resolved: 14/Dec/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Question | Priority: | Major - P3 |
| Reporter: | Tommaso Tocci | Assignee: | [DO NOT ASSIGN] Backlog - Server Development Platform Team (SDP) (Inactive) |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||
| Assigned Teams: |
Server Development Platform
|
||||||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||||||
| Description |
Motivation for RequestReduce considerably core server EVG patch time. ContextAs part of my Skunkwork project I tried to understand why starting up a sharded cluster using ShardingTest takes so long. Surprisingly I discovered that just spawning the mongod processes can take up to ~30 secs on Evegreen. With "spawning" I mean the time that pass from when we [call execve| we call execve] to the time we execute the first instruction in mongod_main.
As of today we run ~883 tests in every of the 10 sharding suites, each test take ~30 secs just to spawn the process required to setup the cluster. So only accounting for sharding tests, we are spending at least 73 AWS machine hours per patch per variant just to spawn those processes. It seems we have a huge space for improvement here. |
| Comments |
| Comment by Andrew Morrow (Inactive) [ 25/Oct/22 ] |
|
We need to do a little thinking about exactly which scenarios admit dropping z,now but I agree we should do this. I also expect that --experimental-optimization=+{vishidden,fnsi} may offer even another 30% reduction in startup times. |
| Comment by Tommaso Tocci [ 25/Oct/22 ] |
|
alexander.neben@mongodb.com I already did some comparison analysis and it seems that lazy symbol resolution would reduce overall EVG patch time. You can analyze the result by comparing: In RHEL 8.0 variant I observed tasks run time reduction that span between 5% and 20% |
| Comment by Tommaso Tocci [ 24/Oct/22 ] |
|
andrew.morrow@mongodb.com removing the -z now linking flag we have a ~30% startup time improvement, I've opened |
| Comment by Alex Neben [ 04/Aug/22 ] |
|
This might fall more on SDP than DAG and there are a ton of implications here. Thank you for getting us this data and I hope there is something we can do about it. |