[SERVER-40779] Disable sys-perf cluster reuse to avoid leaking EC2 instances Created: 23/Apr/19  Updated: 06/Dec/22

Status: Backlog
Project: Core Server
Component/s: Performance
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Backlog - Server Tooling and Methods (STM) (Inactive)
Resolution: Unresolved Votes: 0
Labels: tig-evgconfig
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-33142 Experiment with disabling cluster reu... Closed
Assigned Teams:
Server Tooling & Methods
Participants:

 Description   

We should attempt to re-commit the changes from 5bd4de9 as part of SERVER-33142. Running infrastructure_teardown.py as part of the "post" phase of the Evergreen task should hopefully be more reliable than running the teardown script.

  • We had issues where the Evergreen task is aborted and because the .tfstate file wasn't written to the /data/infrastructure_provisioning/terraform/ directory causing the teardown script to do nothing and leading to EC2 instances being leaked. Note that the infrastructure_provisioning.py script first writes the .tfstate file to the working directory and then copies it over.
  • We've also had issues where the /data drive is unmounted before the teardown script has actually run due to multiple Evergreen host termination jobs running concurrently.

diff --git a/etc/system_perf.yml b/etc/system_perf.yml
index 2d6aa3021db..64037979606 100644
--- a/etc/system_perf.yml
+++ b/etc/system_perf.yml
@@ -42,9 +42,7 @@ post:
         working_dir: work
         script: |
           source ./dsienv.sh
-          if [ -e /data/infrastructure_provisioning/terraform/provisioned.${cluster} ]; then
-            mark_idle.sh
-          fi
+          infrastructure_teardown.py
 
     - command: shell.exec
       params:



 Comments   
Comment by Steven Vannelli [ 10/May/22 ]

Moving this ticket to the Backlog and removing the "Backlog" fixVersion as per our latest policy for using fixVersions.

Comment by David Daly [ 17/Sep/20 ]

I think this is now a duplicate of the completed SERVER-46766. Does that sound right robert.guo?

Generated at Thu Feb 08 04:55:56 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.