[SERVER-59313] SCons cachedir symlink mode Created: 12/Aug/21  Updated: 27/Oct/23  Resolved: 27/Oct/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 6.0 Desired

Type: Improvement Priority: Major - P3
Reporter: Daniel Moody Assignee: [DO NOT ASSIGN] Backlog - Server Development Platform Team (SDP) (Inactive)
Resolution: Won't Do Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Assigned Teams:
Server Development Platform
Participants:

 Description   

This was an idea I had while working on the PCH project and discovering that pch obj files are much larger and there fore slow down the cache due to increased transfer speed. The effects of this idea would be useful even outside of the context of PCH. This would be a special mode, intended for use in evergreen where we transfer a large amount of data over network from the scons cache EFS.

The idea would be to make the cache use a symlink when copying from the cache instead of actually copying the file. There are probably some other places we will need to examine that are able to use the symlinks correctly, for example:

  • Does hardlink install action still work correctly?
  • Do packaging scripts work with these symlinked files?
  • To make sure the symlinked file doesn't drop out from under us while we are linked, we will need to make a hardlink inside the EFS, and symlink to that instead, and then delete the hardlink after the evergreen build is done?


 Comments   
Comment by Daniel Moody [ 16/Aug/21 ]

Stipulation posed by acm:
Reading the symlink will still force the data over the network anyways anytime the file is used, so this might not be really saving much.

possibly if we had cache=all, then in a hot cache situation it would not need to relink the libs and bins, but we would pay for uploading and storing a large amount of data generated from the linked files. Is the trade off worth it? We would probably be relinking a majority of the time in CI so this might not be feasible or useful change.

Generated at Thu Feb 08 05:46:55 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.