[DRIVERS-2657] Make Lambda Tests More Resilient Created: 20/Jun/23  Updated: 21/Aug/23

Status: Implementing
Project: Drivers
Component/s: FaaS
Fix Version/s: None

Type: Spec Change Priority: Major - P3
Reporter: Durran Jordan Assignee: Durran Jordan
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Issue split
split to JAVA-5042 Make Lambda Tests More Resilient Backlog
split to CSHARP-4692 Make Lambda Tests More Resilient Backlog
split to CDRIVER-4670 Make Lambda Tests More Resilient Closed
split to CXX-2704 Make Lambda Tests More Resilient Closed
split to GODRIVER-2880 Make Lambda Tests More Resilient Closed
split to MOTOR-1147 Make Lambda Tests More Resilient Closed
split to NODE-5373 Make Lambda Tests More Resilient Closed
split to PHPLIB-1171 Make Lambda Tests More Resilient Closed
split to PYTHON-3775 Make Lambda Tests More Resilient Closed
split to RUBY-3283 Make Lambda Tests More Resilient Closed
split to RUST-1691 Make Lambda Tests More Resilient Closed
Related
is related to NODE-5361 Flaky Test - Atlas Cluster Limit [1] Closed
Driver Changes: Needed
Downstream Changes Summary:

Drivers that currently automate FaaS testing should update their Evergreen configurations to use task groups instead of the single tools script as described in https://github.com/mongodb/specifications/commit/8debfaebfc20402d26d8058661d51309ba5e2344 .

Engineering Lead: Andreas Braun Andreas Braun
Start date:
Driver Compliance:
Key Status/Resolution FixVersion
CDRIVER-4670 Won't Do
CXX-2704 Won't Do
CSHARP-4692 Backlog
GODRIVER-2880 Duplicate
JAVA-5042 Backlog
NODE-5373 Duplicate
MOTOR-1147 Duplicate
PYTHON-3775 Duplicate
PHPLIB-1171 Won't Do
RUBY-3283 Done
RUST-1691 Duplicate

 Description   

Summary

When the automated AWS Lambda tests error or are aborted, Atlas clusters run the risk of being orphaned. The current script attempts to handle this in a trap, but that is not always 100% effective.

The recommendation is to split the script (https://github.com/mongodb-labs/drivers-evergreen-tools/blob/master/.evergreen/run-deployed-lambda-aws-tests.sh) into separate scripts that can be run in driver CI as task groups where the setup and teardown can be guaranteed regardless of error.

This should/could also be given additional protection by using cloud-dev instead of cloud and setting auto-expiration times much lower that the default, if possible.

Motivation

Who is the affected end user?

Drivers getting orphaned clusters in Atlas due to these tests.

How does this affect the end user?

Can incur high Atlas costs - for example the Node team hit the monthly limit.

How likely is it that this problem or use case will occur?

Regularly

If the problem does occur, what are the consequences and how severe are they?

Once driver team cluster limit is hit, no more clusters can be created and subsequent tests will always fail.

Is this issue urgent?

For Node, yes.

Is this ticket required by a downstream team?

No

Is this ticket only for tests?

Test improvements

Acceptance Criteria

  • Reorganize the lambda scripts in drivers-evergreen-tools
  • Update the spec recommendations to use task groups
  • Show working prototype CI in 1 driver


 Comments   
Comment by Githook User [ 28/Jun/23 ]

Author:

{'name': 'Durran Jordan', 'email': 'durran@gmail.com', 'username': 'durran'}

Message: test(DRIVERS-2657): break out lambda scripts (#325)

  • fix: always allow all cleanup to run on lambda
  • test: update scripts
  • chore: revert old script
  • test: update scripts
  • test: dont create cluster in run script
  • test: check cluster in setup
  • test: allow supplying mongodb version
  • test: check every 15, wait 20 min
  • test: tweak cluster settings
  • test: more config tweaks
  • test: auto scaling required
  • test: add read preference back
  • Update .evergreen/aws_lambda/setup-atlas-cluster.sh

Co-authored-by: Bailey Pearson <bailey.pearson@gmail.com>

  • test: try more expansions
  • test: update suggestions
  • Update .evergreen/aws_lambda/setup-atlas-cluster.sh

Co-authored-by: Andreas Braun <alcaeus@users.noreply.github.com>

  • test: move atlas scripts

---------

Co-authored-by: Bailey Pearson <bailey.pearson@gmail.com>
Co-authored-by: Andreas Braun <alcaeus@users.noreply.github.com>
Branch: master
https://github.com/mongodb-labs/drivers-evergreen-tools/commit/797f71cdd51b30a13a143ee028982c898519e85b

Comment by Githook User [ 28/Jun/23 ]

Author:

{'name': 'Durran Jordan', 'email': 'durran@gmail.com', 'username': 'durran'}

Message: test(DRIVERS-2657): update faas testing spec (#1438)

  • chore: update per suggestions
  • fix: table formatting
Comment by Githook User [ 22/Jun/23 ]

Author:

{'name': 'Durran Jordan', 'email': 'durran@gmail.com', 'username': 'durran'}

Message: test(DRIVERS-2657): break out lambda scripts
Branch: DRIVERS-2657
https://github.com/mongodb-labs/drivers-evergreen-tools/commit/a3899fd3730cafc8fcf8d315554c9ab98bfc252a

Comment by Githook User [ 22/Jun/23 ]

Author:

{'name': 'Durran Jordan', 'email': 'durran@gmail.com', 'username': 'durran'}

Message: test(DRIVERS-2657): break out lambda scripts
Branch: DRIVERS-2657
https://github.com/mongodb-labs/drivers-evergreen-tools/commit/dfa97b1abf46d22c16d17a1a925073bc94d45a3f

Generated at Thu Feb 08 08:26:07 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.