-
Type: Spec Change
-
Resolution: Unresolved
-
Priority: Major - P3
-
None
-
Component/s: FaaS
-
None
-
Needed
-
Summary
When the automated AWS Lambda tests error or are aborted, Atlas clusters run the risk of being orphaned. The current script attempts to handle this in a trap, but that is not always 100% effective.
The recommendation is to split the script (https://github.com/mongodb-labs/drivers-evergreen-tools/blob/master/.evergreen/run-deployed-lambda-aws-tests.sh) into separate scripts that can be run in driver CI as task groups where the setup and teardown can be guaranteed regardless of error.
This should/could also be given additional protection by using cloud-dev instead of cloud and setting auto-expiration times much lower that the default, if possible.
Motivation
Who is the affected end user?
Drivers getting orphaned clusters in Atlas due to these tests.
How does this affect the end user?
Can incur high Atlas costs - for example the Node team hit the monthly limit.
How likely is it that this problem or use case will occur?
Regularly
If the problem does occur, what are the consequences and how severe are they?
Once driver team cluster limit is hit, no more clusters can be created and subsequent tests will always fail.
Is this issue urgent?
For Node, yes.
Is this ticket required by a downstream team?
No
Is this ticket only for tests?
Test improvements
Acceptance Criteria
- Reorganize the lambda scripts in drivers-evergreen-tools
- Update the spec recommendations to use task groups
- Show working prototype CI in 1 driver
- is related to
-
NODE-5361 Flaky Test - Atlas Cluster Limit [1]
- Closed
- split to
-
JAVA-5042 Make Lambda Tests More Resilient
- Closed
-
CDRIVER-4670 Make Lambda Tests More Resilient
- Closed
-
CSHARP-4692 Make Lambda Tests More Resilient
- Closed
-
CXX-2704 Make Lambda Tests More Resilient
- Closed
-
GODRIVER-2880 Make Lambda Tests More Resilient
- Closed
-
MOTOR-1147 Make Lambda Tests More Resilient
- Closed
-
NODE-5373 Make Lambda Tests More Resilient
- Closed
-
PHPLIB-1171 Make Lambda Tests More Resilient
- Closed
-
PYTHON-3775 Make Lambda Tests More Resilient
- Closed
-
RUBY-3283 Make Lambda Tests More Resilient
- Closed
-
RUST-1691 Make Lambda Tests More Resilient
- Closed