Uploaded image for project: 'Evergreen'
  1. Evergreen
  2. EVG-16247

Add capability for "singleton tasks" which only run once

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Debugging
    • Major - P3
    • Resolution: Unresolved
    • None
    • None
    • product
    • None

    Description

      In SERVER-63191, we identified a need to gather data relevant for an entire build which may change during the build run and yet which must remain static for the entire build. The example we have in SERVER-63191 is version metadata (git tags), but there are other potential pieces of data that fit this description as well (python/go/npm module versions, for example). For consistency, this data needs to be collected only once during a build and then persisted via expansions to all build variants and tasks.

      This need has come up a couple of times recently in separate projects (the other one is toolchain-builder. While we can implement the desired behavior via evergreen.yml and some scripting, this seems like a feature that should exist within Evergreen. Is this something we can get implemented? We were thinking this might look like a task that all other tasks can depend on to complete and which gets run only once for an entire build across all build variants. So, for instance, all build variants can depend on a task that runs a script to fetch external data for expansions and then uploads it to S3, but only one of those build variants will actually run the task. A task dependency might look like this, for example:

      singleton_tasks:
        - name: create_global_expansions
          depends_on: []
          commands:
            ...
       
      - name: compile
        tasks:
          -name: create_global_expansions
          -name: fetch_global_expansions
      

      In the above, create_global_expansions would run once for the entire build, but fetch_global_expansions would run once per build variant.

      NB: the way we were going to do this on our own was to add a task which all build variants run first to conditionally generate the data for the expansions and then upload it to S3. If the data was already in S3, the task would simply download it and update expansions. This seemed hacky and still allowed racing, which is why we thought this would be better as an Evergreen feature request, and having to ask Evergreen whether a task has run makes this seem like a missing feature in Evergreen. Also, we expected to run into interactions with generated tasks as well, which we would like to avoid if at all possible.

      Attachments

        Issue Links

          Activity

            People

              cristopher.stauffer@mongodb.com Cristopher Stauffer
              ryan.egesdahl@mongodb.com Ryan Egesdahl
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated: