Uploaded image for project: 'Evergreen'
  1. Evergreen
  2. EVG-694

Evergreen commands should surface as shell interfaces



    • Type: New Feature
    • Status: Resolved
    • Priority: Major - P3
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: app
    • Labels:


      There's been a longstanding request to add control flow (if, for, while, etc) to the Evergreen configuration format. This is definitely something we need to address, but I am very wary about making the config format Turing complete. I'll let Google's awesome Kubernetes retrospective do the talking for me:

      ...configuration-management systems tend to invent a domain-specific configuration language that (eventually) becomes Turing complete, starting from the desire to perform computation on the data in the configuration (e.g., to adjust the amount of memory to give a server as a function of the number of shards in the service). The result is the kind of inscrutable "configuration is code" that people were trying to avoid by eliminating hard-coded parameters in the application's source code. It doesn't reduce operational complexity or make the configurations easier to debug or change; it just moves the computations from a real programming language to a domain-specific one, which typically has weaker development tools (e.g., debuggers, unit test frameworks, etc).

      We believe the most effective approach is to accept this need, embrace the inevitability of programmatic configuration, and maintain a clean separation between computation and data. The language to represent the data should be a simple, data-only format such as JSON or YAML, and programmatic modification of this data should be done in a real programming language, where there are well-understood semantics, as well as good tooling. Interestingly, this same separation of computation and data can be seen in the disparate field of front-end development with frameworks such as Angular that maintain a crisp separation between the worlds of markup (data) and JavaScript (computation).

      I think the solution that will ultimately require the least amount of debugging for both Evergreen developers and Evergreen users may be to offload task scripting to a true scripting language--perhaps we could just use the shell?

      What if Evergreen commands were surfaced as executable commands that were brought into PATH? So instead of a complicated set of YAML commands, everything could be a shell command that hooks into Evergreen's provided features. Something like

        "upload tool":
          command: s3.put
            aws_key: ${aws_key}
            aws_secret: ${aws_secret}
            local_file: src/bin/${tool}
            remote_file: mongo-tools/binaries/${build_id}/${edition|community}/${tool}${extension}
            bucket: mciuploads
            permissions: public-read
            content_type: application/octet-stream
            display_name: ${tool}

      could be written into a shell script as

      evg s3.put --local_file src/bin/${tool}\
       --remote_file mongo-tools/binaries/${build_id}/${edition|community}/${tool}${extension} \
       --bucket mciuploads --permissions public-read \
       --content_type application/octet-stream --display_name ${tool}

      which would allow us to easily do control flow around commands without writing and maintaining our own evergreen language. I'd imagine the "evg" binary would just be a mode for the agent binary that acted as a simple interface to send HTTP commands to the real agent process.

      These commands wouldn't necessarily only be accessible through bash. Since they are just calls to a binary executable, they could be wrapped by practically any interpreted language.


          Issue Links



              backlog-server-evg Backlog - Evergreen Team
              kyle.erf Kyle Erf
              2 Vote for this issue
              7 Start watching this issue