[DRIVERS-2657] Make Lambda Tests More Resilient Created: 20/Jun/23 Updated: 21/Aug/23 |
|
| Status: | Implementing |
| Project: | Drivers |
| Component/s: | FaaS |
| Fix Version/s: | None |
| Type: | Spec Change | Priority: | Major - P3 |
| Reporter: | Durran Jordan | Assignee: | Durran Jordan |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Driver Changes: | Needed | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Downstream Changes Summary: | Drivers that currently automate FaaS testing should update their Evergreen configurations to use task groups instead of the single tools script as described in https://github.com/mongodb/specifications/commit/8debfaebfc20402d26d8058661d51309ba5e2344 . |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Engineering Lead: | |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Start date: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Driver Compliance: |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description |
SummaryWhen the automated AWS Lambda tests error or are aborted, Atlas clusters run the risk of being orphaned. The current script attempts to handle this in a trap, but that is not always 100% effective. The recommendation is to split the script (https://github.com/mongodb-labs/drivers-evergreen-tools/blob/master/.evergreen/run-deployed-lambda-aws-tests.sh) into separate scripts that can be run in driver CI as task groups where the setup and teardown can be guaranteed regardless of error. This should/could also be given additional protection by using cloud-dev instead of cloud and setting auto-expiration times much lower that the default, if possible. MotivationWho is the affected end user?Drivers getting orphaned clusters in Atlas due to these tests. How does this affect the end user?Can incur high Atlas costs - for example the Node team hit the monthly limit. How likely is it that this problem or use case will occur?Regularly If the problem does occur, what are the consequences and how severe are they?Once driver team cluster limit is hit, no more clusters can be created and subsequent tests will always fail. Is this issue urgent?For Node, yes. Is this ticket required by a downstream team?No Is this ticket only for tests?Test improvements Acceptance Criteria
|
| Comments |
| Comment by Githook User [ 28/Jun/23 ] |
|
Author: {'name': 'Durran Jordan', 'email': 'durran@gmail.com', 'username': 'durran'}Message: test(DRIVERS-2657): break out lambda scripts (#325)
Co-authored-by: Bailey Pearson <bailey.pearson@gmail.com>
Co-authored-by: Andreas Braun <alcaeus@users.noreply.github.com>
--------- Co-authored-by: Bailey Pearson <bailey.pearson@gmail.com> |
| Comment by Githook User [ 28/Jun/23 ] |
|
Author: {'name': 'Durran Jordan', 'email': 'durran@gmail.com', 'username': 'durran'}Message: test(DRIVERS-2657): update faas testing spec (#1438)
|
| Comment by Githook User [ 22/Jun/23 ] |
|
Author: {'name': 'Durran Jordan', 'email': 'durran@gmail.com', 'username': 'durran'}Message: test(DRIVERS-2657): break out lambda scripts |
| Comment by Githook User [ 22/Jun/23 ] |
|
Author: {'name': 'Durran Jordan', 'email': 'durran@gmail.com', 'username': 'durran'}Message: test(DRIVERS-2657): break out lambda scripts |