[DRIVERS-2585] Use AWS Secrets Manager for AWS-Related Test Secrets Created: 28/Mar/23  Updated: 29/Jan/24

Status: Scheduled
Project: Drivers
Component/s: None
Fix Version/s: None

Type: Epic Priority: Unknown
Reporter: Steve Silvester Assignee: Noah Stapp
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Issue split
split to CDRIVER-4701 Use AWS Secrets Manager for AWS-Relat... Blocked
split to CSHARP-4741 Use AWS Secrets Manager for AWS-Relat... Blocked
split to CXX-2724 Use AWS Secrets Manager for AWS-Relat... Blocked
split to GODRIVER-2928 Use AWS Secrets Manager for AWS-Relat... Blocked
split to JAVA-5094 Use AWS Secrets Manager for AWS-Relat... Blocked
split to MOTOR-1167 Use AWS Secrets Manager for AWS-Relat... Blocked
split to NODE-5507 Use AWS Secrets Manager for AWS-Relat... Blocked
split to PHPLIB-1216 Use AWS Secrets Manager for AWS-Relat... Blocked
split to PYTHON-3895 Use AWS Secrets Manager for AWS-Relat... Blocked
split to RUBY-3311 Use AWS Secrets Manager for AWS-Relat... Blocked
split to RUST-1717 Use AWS Secrets Manager for AWS-Relat... Blocked
Related
is related to DRIVERS-2415 Implement OIDC SASL mechanism Implementing
Driver Changes: Needed
Engineering Lead: Steven Silvester Steven Silvester
Program Manager: Esha Bhargava Esha Bhargava
Scope Cost Estimate: 0
Cost to Date: 0
Final Cost Estimate: 0
Cost Threshold %: 100
Detailed Project Statuses:

Engineer(s): Noah Stapp
Summary: Migrate AWS Secrets to AWS Secret Manager from Evergreen Project Variables.

2023-09-15:

  • Status update:
    • Completed AWS tests with the Python driver.
    • Paused work to focus on other quarterly tasks.

2023-09-01:

  • Status update:
    • Finished Atlas connection tests, wrapping up AWS tests with Python Driver.  Go Driver has implemented Atlas connection tests.  OIDC is also being migrated as part of DRIVERS-2415 updates this quarter.
  • Risks or delays:
    • Some secret values may need to be re-generated if the original source is lost
    • Variations in Evergreen project configuration have required additional work to generalize AWS Secret integrations.

2023-08-21:

  • Status update:
    • First implementation in Python underway, steadily progressing through test suites.
    • Separating each test suite's secrets into separate vaults for better security. 
  • Risks or delays:
    • Some secret values may need to be re-generated if the original source is lost
    • Possible variations in Evergreen project configuration could require additional work to generalize AWS Secret integrations.
Driver Compliance:
Key Status/Resolution FixVersion
CDRIVER-4701 Blocked
CXX-2724 Blocked
CSHARP-4741 Blocked
GODRIVER-2928 Blocked
JAVA-5094 Blocked
NODE-5507 Blocked
MOTOR-1167 Blocked
PYTHON-3895 Blocked
PHPLIB-1216 Blocked
RUBY-3311 Blocked
RUST-1717 Blocked

 Description   

Summary

_We currently have around 20 Evergreen Project variables that are used to populate a ${DRIVERS_TOOLS}/.evergreen/auth_aws/aws_e2e_setup.json file that is used in Drivers Evergeen Tools in test scripts. As part of DRIVERS-2415, we now have a mechanism to store and retrieve variables using AWS Secrets Manager, rather than continuing to grow this list of manually updated variables across all drivers.

All members of dbx have access to view and update the secrets using the drivers-test-secrets-role login option in the Drivers AWS account.

This project would move the existing affected Project Variables and create a new wiki page for the maintenance and upkeep of these secrets. There would be a new script created in Drivers Evergreen Tools to create an expansion file used by EG to provide these values as environment variables which can then be used by the existing scripts instead of loading the values from aws_e2e_setup.json.

Drivers would then replace the portion of their Evergreen Config with a block that acquires the appropriate credentials and expands the variables. They would also be able to remove the affected project variables from EG.

Motivation

Adding and updating credentials currently requires the coordination of all of the driver teams, and manual effort.



 Comments   
Comment by Githook User [ 29/Jan/24 ]

Author:

{'name': 'Steven Silvester', 'email': 'steven.silvester@ieee.org', 'username': 'blink1073'}

Message: DRIVERS-2585 Use AWS Secrets Manager for CSFLE (#390)

  • Add csfle scripts

---------

Co-authored-by: Kevin Albertson <kevin.albertson@10gen.com>
Branch: master
https://github.com/mongodb-labs/drivers-evergreen-tools/commit/4393c2873d25dceaacd9ec27278d3b795e82c933

Comment by Githook User [ 01/Sep/23 ]

Author:

{'name': 'Steven Silvester', 'email': 'steven.silvester@ieee.org', 'username': 'blink1073'}

Message: DRIVERS-2585 Make the Secrets Access errors more user friendly (#347)

DRIVERS-2585 Make the errors more user friendly
Branch: master
https://github.com/mongodb-labs/drivers-evergreen-tools/commit/98968d3ed3015fbe68d63616a558da2817d8173b

Comment by Githook User [ 30/Aug/23 ]

Author:

{'name': 'Steven Silvester', 'email': 'steven.silvester@ieee.org', 'username': 'blink1073'}

Message: DRIVERS-2585 Migrate OIDC Secrets Handling (#345)
Branch: master
https://github.com/mongodb-labs/drivers-evergreen-tools/commit/da3c4267f1f9065592c1693f75ed9f1b2792cce0

Comment by Githook User [ 14/Aug/23 ]

Author:

{'name': 'Noah Stapp', 'email': 'noah.stapp@mongodb.com', 'username': 'NoahStapp'}

Message: DRIVERS-2585 Default to AWS_PROFILE if a profile is not provided (#337)
Branch: master
https://github.com/mongodb-labs/drivers-evergreen-tools/commit/efbd141ca717f82f2ab18ac78855d008a0d8e63c

Comment by Githook User [ 08/Aug/23 ]

Author:

{'name': 'Noah Stapp', 'email': 'noah.stapp@mongodb.com', 'username': 'NoahStapp'}

Message: DRIVERS-2585 Use AWS Secrets Manager for AWS-Related Test Secrets (#334)
Branch: master
https://github.com/mongodb-labs/drivers-evergreen-tools/commit/0361b3fd3a9f1641518aa2f37d872d3346e8a450

Comment by Tom Selander [ 18/Apr/23 ]

Bringing this to triage today since this came up again in Slack for where we store secrets. We should figure out next steps for 1Password even if this ticket doesn't get picked up

Generated at Thu Feb 08 08:25:56 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.