[SERVER-54232] Use install rather than cp to copy files in buildsystem Created: 03/Feb/21 Updated: 27/Oct/23 Resolved: 27/Oct/23 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Build |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Mathias Stearn | Assignee: | [DO NOT ASSIGN] Backlog - Server Development Platform Team (SDP) (Inactive) |
| Resolution: | Won't Do | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Assigned Teams: |
Server Development Platform
|
||||||||
| Participants: | |||||||||
| Description |
|
One of the weird things I learned while working on ninja is that install is significantly faster than cp. I don't know why this is, but it is reproducible across many systems and filesystems, etc. For example, this is running on a virtual workstation (c5.4xl ubuntu 18.04 xfs):
I have tried on my laptop (WSL2 + Arch ext4 on nvme) and the results were similar. Including the weirdness of later runs of cp being slower than earlier runs. � |
| Comments |
| Comment by Andrew Morrow (Inactive) [ 03/Feb/21 ] |
|
redbeard0531 - Once we have something that works at all layers with --install-action, then we can make the default better, and potentially Ninja aware, so it won't need to be opt in. |
| Comment by Mathias Stearn [ 03/Feb/21 ] |
|
I'm fine considering it part of |
| Comment by Daniel Moody [ 03/Feb/21 ] |
|
Well the CR out for |
| Comment by Andrew Morrow (Inactive) [ 03/Feb/21 ] |
|
redbeard0531 - Ah right, I remember when we investigated that. OK, well, still think |
| Comment by Mathias Stearn [ 03/Feb/21 ] |
|
Ninja does not remove outputs prior to running commands, otherwise that would break things like restat=1. This is also why you need to remove static libs in the AR rule, because someone decided it would be a good idea for ar to be stateful: https://github.com/mongodb/mongo/blob/26a3e7bc2a773b992c51808577ae999fc0e66d0f/site_scons/site_tools/ninja.py#L489-L498. |
| Comment by Andrew Morrow (Inactive) [ 03/Feb/21 ] |
|
redbeard0531 - We are working in CC daniel.moody |
| Comment by Mathias Stearn [ 03/Feb/21 ] |
|
I figured out why install is faster at https://superuser.com/a/229980. It removes the file then replaces it, whereas cp truncates and appends in place. Which means that not only is install faster, it is also probably more correct since it won't crash a running program if it is loaded. I verified that cp --remove-dest performs as well as install, but I think it is a gnu extension so it may not work everywhere. |