Details
-
Task
-
Status: Backlog
-
Major - P3
-
Resolution: Unresolved
-
None
-
None
-
Server Tooling & Methods
Description
We should attempt to re-commit the changes from 5bd4de9 as part of SERVER-33142. Running infrastructure_teardown.py as part of the "post" phase of the Evergreen task should hopefully be more reliable than running the teardown script.
- We had issues where the Evergreen task is aborted and because the .tfstate file wasn't written to the /data/infrastructure_provisioning/terraform/ directory causing the teardown script to do nothing and leading to EC2 instances being leaked. Note that the infrastructure_provisioning.py script first writes the .tfstate file to the working directory and then copies it over.
- We've also had issues where the /data drive is unmounted before the teardown script has actually run due to multiple Evergreen host termination jobs running concurrently.
diff --git a/etc/system_perf.yml b/etc/system_perf.yml
|
index 2d6aa3021db..64037979606 100644
|
--- a/etc/system_perf.yml
|
+++ b/etc/system_perf.yml
|
@@ -42,9 +42,7 @@ post:
|
working_dir: work
|
script: |
|
source ./dsienv.sh
|
- if [ -e /data/infrastructure_provisioning/terraform/provisioned.${cluster} ]; then
|
- mark_idle.sh
|
- fi
|
+ infrastructure_teardown.py
|
|
- command: shell.exec
|
params: |
Attachments
Issue Links
- is related to
-
SERVER-33142 Experiment with disabling cluster reuse in system_perf.yml
-
- Closed
-