Uploaded image for project: 'Drivers'
  1. Drivers
  2. DRIVERS-2585

Use AWS Secrets Manager for Evergreen Test Secrets

XMLWordPrintableJSON

    • Type: Icon: Epic Epic
    • Resolution: Unresolved
    • Priority: Icon: Unknown Unknown
    • None
    • Component/s: None
    • None
    • Hide

      Summary of necessary driver changes

      Show
      Summary of necessary driver changes See instructions in Drivers Evergreen Tools for usage of the vaults, and the READMEs in each of the sub-folders that use secrets. Drivers are expected to migrate ALL of the secrets listed in the Secrets Handling README. See https://github.com/mongodb/mongo-go-driver/commit/9755f4aa7c303ba3b79d16e28ddefe533b043f23 for an example using the scripts to handle secrets instead of EVG project variables. See https://github.com/mongodb/mongo-go-driver/commit/e77ab7041f86e65ecc665ffe5c7b3f350eb0a741 for an example of migrating CSFLE to use the new scripts. See https://github.com/mongodb/mongo-go-driver/commit/70b1fa64ef2999635ab843ae7d3a6b2dde07e118 for an example of migrating AWS tests to use the new scripts.
    • To Do
    • Use AWS Secrets Manager for AWS-Related Test Secrets
    • 0
    • 0
    • 0
    • 100
    • Hide

      Engineer(s): Noah Stapp
      Summary: Migrate AWS Secrets to AWS Secret Manager from Evergreen Project Variables.

      2023-09-15:

      • Status update:
        • Completed AWS tests with the Python driver.
        • Paused work to focus on other quarterly tasks.

      2023-09-01:

      • Status update:
        • Finished Atlas connection tests, wrapping up AWS tests with Python Driver.  Go Driver has implemented Atlas connection tests.  OIDC is also being migrated as part of DRIVERS-2415 updates this quarter.
      • Risks or delays:
        • Some secret values may need to be re-generated if the original source is lost
        • Variations in Evergreen project configuration have required additional work to generalize AWS Secret integrations.

      2023-08-21:

      • Status update:
        • First implementation in Python underway, steadily progressing through test suites.
        • Separating each test suite's secrets into separate vaults for better security. 
      • Risks or delays:
        • Some secret values may need to be re-generated if the original source is lost
        • Possible variations in Evergreen project configuration could require additional work to generalize AWS Secret integrations.
      Show
      Engineer(s): Noah Stapp Summary:  Migrate AWS Secrets to AWS Secret Manager from Evergreen Project Variables. 2023-09-15 : Status update: Completed AWS tests with the Python driver. Paused work to focus on other quarterly tasks. 2023-09-01 : Status update: Finished Atlas connection tests, wrapping up AWS tests with Python Driver.  Go Driver has implemented Atlas connection tests.  OIDC is also being migrated as part of DRIVERS-2415 updates this quarter. Risks or delays: Some secret values may need to be re-generated if the original source is lost Variations in Evergreen project configuration have required additional work to generalize AWS Secret integrations. 2023-08-21 : Status update: First implementation in Python underway, steadily progressing through test suites. Separating each test suite's secrets into separate vaults for better security.  Risks or delays: Some secret values may need to be re-generated if the original source is lost Possible variations in Evergreen project configuration could require additional work to generalize AWS Secret integrations.
    • Needed
    • $i18n.getText("admin.common.words.hide")
      Key Status/Resolution FixVersion
      CDRIVER-4701 Backlog
      CXX-2724 Backlog
      CSHARP-4741 In Progress
      GODRIVER-2928 Fixed 2.0.0
      JAVA-5094 Ready for Work
      NODE-5507 Backlog
      MOTOR-1167 Done
      PYTHON-3895 Fixed 4.7
      RUBY-3311 Backlog
      RUST-1717 Backlog
      PHPLIB-1216 Backlog
      PHPC-2395 Fixed 1.20.0
      $i18n.getText("admin.common.words.show")
      #scriptField, #scriptField *{ border: 1px solid black; } #scriptField{ border-collapse: collapse; } #scriptField td { text-align: center; /* Center-align text in table cells */ } #scriptField td.key { text-align: left; /* Left-align text in the Key column */ } #scriptField a { text-decoration: none; /* Remove underlines from links */ border: none; /* Remove border from links */ } /* Add green background color to cells with FixVersion */ #scriptField td.hasFixVersion { background-color: #00FF00; /* Green color code */ } /* Center-align the first row headers */ #scriptField th { text-align: center; } Key Status/Resolution FixVersion CDRIVER-4701 Backlog CXX-2724 Backlog CSHARP-4741 In Progress GODRIVER-2928 Fixed 2.0.0 JAVA-5094 Ready for Work NODE-5507 Backlog MOTOR-1167 Done PYTHON-3895 Fixed 4.7 RUBY-3311 Backlog RUST-1717 Backlog PHPLIB-1216 Backlog PHPC-2395 Fixed 1.20.0

      Summary

      We currently have around 20 Evergreen Project variables that are used to populate a ${DRIVERS_TOOLS}/.evergreen/auth_aws/aws_e2e_setup.json file that is used in Drivers Evergreen Tools in test scripts. Additionally, there are many other secrets used by Drivers Evergreen Tools such as in .evergreen/atlas to launch Atlas clusters.

      As part of DRIVERS-2415, we now have a mechanism to store and retrieve variables using AWS Secrets Manager, rather than continuing to grow this list of manually updated variables across all drivers.

      All members of dbx have access to view and update the secrets using the drivers-test-secrets-role login option in the Drivers AWS account.

      This project would move the existing affected Project Variables and create a new wiki page for the maintenance and upkeep of these secrets. There would be a new script created in Drivers Evergreen Tools to create an expansion file used by EG to provide these values as environment variables which can then be used by the existing scripts.

      Drivers would then replace the portion of their Evergreen Config with a block that acquires the appropriate credentials and expands the variables. They would also be able to remove the affected project variables from EG.

      Motivation

      Adding and updating credentials currently requires the coordination of all of the driver teams, and manual effort.

            Assignee:
            noah.stapp@mongodb.com Noah Stapp
            Reporter:
            steve.silvester@mongodb.com Steve Silvester
            Steven Silvester Steven Silvester
            Esha Bhargava Esha Bhargava
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: