Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-40779

Disable sys-perf cluster reuse to avoid leaking EC2 instances

    XMLWordPrintableJSON

Details

    • Task
    • Status: Backlog
    • Major - P3
    • Resolution: Unresolved
    • None
    • None
    • Performance
    • Server Tooling & Methods

    Description

      We should attempt to re-commit the changes from 5bd4de9 as part of SERVER-33142. Running infrastructure_teardown.py as part of the "post" phase of the Evergreen task should hopefully be more reliable than running the teardown script.

      • We had issues where the Evergreen task is aborted and because the .tfstate file wasn't written to the /data/infrastructure_provisioning/terraform/ directory causing the teardown script to do nothing and leading to EC2 instances being leaked. Note that the infrastructure_provisioning.py script first writes the .tfstate file to the working directory and then copies it over.
      • We've also had issues where the /data drive is unmounted before the teardown script has actually run due to multiple Evergreen host termination jobs running concurrently.

      diff --git a/etc/system_perf.yml b/etc/system_perf.yml
      index 2d6aa3021db..64037979606 100644
      --- a/etc/system_perf.yml
      +++ b/etc/system_perf.yml
      @@ -42,9 +42,7 @@ post:
               working_dir: work
               script: |
                 source ./dsienv.sh
      -          if [ -e /data/infrastructure_provisioning/terraform/provisioned.${cluster} ]; then
      -            mark_idle.sh
      -          fi
      +          infrastructure_teardown.py
       
           - command: shell.exec
             params:
      

      Attachments

        Issue Links

          Activity

            People

              backlog-server-stm Backlog - Server Tooling and Methods (STM) (Inactive)
              max.hirschhorn@mongodb.com Max Hirschhorn
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated: