Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-40779

Disable sys-perf cluster reuse to avoid leaking EC2 instances

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Open
    • Priority: Major - P3
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: Backlog
    • Component/s: Performance
    • Labels:

      Description

      We should attempt to re-commit the changes from 5bd4de9 as part of SERVER-33142. Running infrastructure_teardown.py as part of the "post" phase of the Evergreen task should hopefully be more reliable than running the teardown script.

      • We had issues where the Evergreen task is aborted and because the .tfstate file wasn't written to the /data/infrastructure_provisioning/terraform/ directory causing the teardown script to do nothing and leading to EC2 instances being leaked. Note that the infrastructure_provisioning.py script first writes the .tfstate file to the working directory and then copies it over.
      • We've also had issues where the /data drive is unmounted before the teardown script has actually run due to multiple Evergreen host termination jobs running concurrently.

      diff --git a/etc/system_perf.yml b/etc/system_perf.yml
      index 2d6aa3021db..64037979606 100644
      --- a/etc/system_perf.yml
      +++ b/etc/system_perf.yml
      @@ -42,9 +42,7 @@ post:
               working_dir: work
               script: |
                 source ./dsienv.sh
      -          if [ -e /data/infrastructure_provisioning/terraform/provisioned.${cluster} ]; then
      -            mark_idle.sh
      -          fi
      +          infrastructure_teardown.py
       
           - command: shell.exec
             params:
      

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: