Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-40779

Disable sys-perf cluster reuse to avoid leaking EC2 instances

    • Type: Icon: Task Task
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Performance
    • Labels:
    • Server Tooling & Methods

      We should attempt to re-commit the changes from 5bd4de9 as part of SERVER-33142. Running infrastructure_teardown.py as part of the "post" phase of the Evergreen task should hopefully be more reliable than running the teardown script.

      • We had issues where the Evergreen task is aborted and because the .tfstate file wasn't written to the /data/infrastructure_provisioning/terraform/ directory causing the teardown script to do nothing and leading to EC2 instances being leaked. Note that the infrastructure_provisioning.py script first writes the .tfstate file to the working directory and then copies it over.
      • We've also had issues where the /data drive is unmounted before the teardown script has actually run due to multiple Evergreen host termination jobs running concurrently.
      Unable to find source-code formatter for language: diff. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
      diff --git a/etc/system_perf.yml b/etc/system_perf.yml
      index 2d6aa3021db..64037979606 100644
      --- a/etc/system_perf.yml
      +++ b/etc/system_perf.yml
      @@ -42,9 +42,7 @@ post:
               working_dir: work
               script: |
                 source ./dsienv.sh
      -          if [ -e /data/infrastructure_provisioning/terraform/provisioned.${cluster} ]; then
      -            mark_idle.sh
      -          fi
      +          infrastructure_teardown.py
           - command: shell.exec

            backlog-server-stm Backlog - Server Tooling and Methods (STM) (Inactive)
            max.hirschhorn@mongodb.com Max Hirschhorn
            0 Vote for this issue
            3 Start watching this issue