Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-8538

Reduce the amount of duplicated artifacts in Evergreen jobs

    • Type: Icon: Improvement Improvement
    • Resolution: Unresolved
    • Priority: Icon: Minor - P4 Minor - P4
    • None
    • Affects Version/s: None
    • Component/s: None
    • Labels:
      None

      Every Evergreen task uploads the full wiredtiger source directory as an artifact, including compiled binaries, but because these tests are using the binaries built in the compile stage we already have a copy of them saved in the compile task's artifact page. This results in needless duplication of artifact files that we need to store and process in Evergreen.

      If we instead only uploaded the files produced by the tests (for example WT_TEST folders) and provided a symlink to the original compile artefacts we could save a large amount of storage space and CPU time spent on this duplicated effort. For the Ubuntu 20.04 build variant approximately 880MB of space and 2 minutes of CPU time, which across 100 tasks comes out to 80GB and 3.5 hours of CPU per patch build.

       

      There may be some cases where we still want to upload the entire folder. For example when a test fails it may be easier to have the entire folder for analysing.

       

      Definition of Done:
      In Evergreen individual tests no longer upload an entire .tgz of the wiredtiger folder, and only upload the delta between their wiredtiger folder and the original compile folder they originally copied.

            Assignee:
            backlog-server-storage-engines [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            andrew.morton@mongodb.com Andrew Morton
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: