[DRIVERS-2377] Add support for GCP attached service accounts when using GCP KMS Created: 09/Aug/21  Updated: 14/Jul/23  Resolved: 14/Jul/23

Status: Closed
Project: Drivers
Component/s: Client Side Encryption
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: William Chow (Inactive) Assignee: Kevin Albertson
Resolution: Done Votes: 2
Labels: MDBW23, size-medium
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on MONGOCRYPT-461 Support `accessToken` for `gcp` KMS p... Closed
is depended on by GODRIVER-2375 Support automatic Authentication for ... Closed
Documented
Issue split
split to JAVA-4685 Add support for GCP attached service ... Closed
split to CDRIVER-4435 Add support for GCP attached service ... Closed
split to CSHARP-4266 Add support for GCP attached service ... Closed
split to CXX-2551 Add support for GCP attached service ... Closed
split to GODRIVER-2501 Add support for GCP attached service ... Closed
split to MOTOR-999 Add support for GCP attached service ... Closed
split to NODE-4462 Add support for GCP attached service ... Closed
split to PHPLIB-917 Add support for GCP attached service ... Closed
split to PYTHON-3367 Add support for GCP attached service ... Closed
split to RUBY-3062 Add support for GCP attached service ... Closed
split to RUST-1417 Add support for GCP attached service ... Closed
Problem/Incident
Related
related to GODRIVER-2415 KMSProvider for GCP does not accept a... Closed
is related to DRIVERS-2280 Obtain AWS credentials for CSFLE in t... Closed
Driver Changes: Needed
Quarter: FY23Q2, FY23Q3, FY24Q1
Downstream Changes Summary:

Summary of required changes

  • Upgrade dependency on libmongocrypt to 1.6.0 or higher. Binaries for 1.6.0 are available on the upload-all task.
  • Call mongocrypt_setopt_use_need_kms_credentials_state to opt in to handling the new MONGOCRYPT_CTX_NEED_KMS_CREDENTIALS state.
  • Handle the new MONGOCRYPT_CTX_NEED_KMS_CREDENTIALS state. If the originally configured KMS providers have an empty gcp: {}, attempt to obtain GCP credentials by sending an HTTP request described in the specification. Pass the new credentials back with mongocrypt_ctx_provide_kms_providers.
  • Add an integration test with a Google Compute Engine (GCE) instance. Get credentials from DRIVERS-2377 test credentials.

Additional background

Please see https://github.com/mongodb/specifications/commit/847d9ba741201f9c9d1305831a9c60e8ab2a1544 for the specification change.

Please see https://github.com/mongodb/mongo-go-driver/commit/91b240c6aab86680ed5e78746a5a5edcd408c237 for a reference implementation in Go.

Consider using the mock server for local development to test the HTTP request to the Metadata Server.

GCP access token is not cached. See the scope for rationale.

Integration test

Drivers are expected to run an integration test with a temporary Google Compute Engine instance. Scripts in the drivers-evergreen-tools .evergreen/csfle/gcpkms directory may be used.

To test, add an Evergreen task group to do the following:

  • Create a GCE instance in a setup_group.
  • Destroy the GCE instance in a teardown_group. Using a teardown_group will destroy the instance if the task fails.

Add a task in the task group to do the following:

  • Build and copy files to the remote GCE instance.
  • Install necessary dependencies on the remote GCE instance.
  • Run the test remotely.

Please see https://github.com/mongodb/mongo-go-driver/commit/91b240c6aab86680ed5e78746a5a5edcd408c237#diff-2bc841e86ce96b7b422ae203fd8315d0b2a461956cecbe0e096420656fc3fb12R2248 for a reference implementation of the integration test in Go.

It may be helpful to refer to driver tests for MONGODB-AWS ECS. The ECS tests perform a similar flow (copying and running a test on a remote ECS instance).

Case:
Driver Compliance:
Key Status/Resolution FixVersion
CDRIVER-4435 Fixed 1.24.0
CXX-2551 Works as Designed 3.8.0
CSHARP-4266 Fixed 2.18.0
GODRIVER-2501 Fixed 1.11.0
JAVA-4685 Fixed 4.8.0
NODE-4462 Fixed 5.1.0
MOTOR-999 Won't Do
PYTHON-3367 Fixed pymongocrypt-1.4, 4.3.3
PHPLIB-917 Done
RUBY-3062 Fixed 2.19.0
RUST-1417 Done 2.6.0
SWIFT-1608 Won't Do

 Description   

We have a customer on GCP, who is trying to use GCP KMS for the CMK and we require a service account key, where an email and privateKey is provided for the service account, in order to use the GCP KMS API. They are using an attached service account according to GCP best practices which says "Use attached service accounts when possible. For applications deployed on Google Cloud that need to use a service account, attach the service account to the underlying compute resource. By attaching a service account, you enable the application to obtain tokens for the service account and to use these tokens to access Google Cloud APIs and resources." and it also says "Use service account keys only if there is no viable alternative". Google even displays the warning "Service account keys could pose a security risk if compromised" when creating a key for a service account. 

The customer is requesting that we follow GCP best practices and when using an "attached" service account, that we skip/bypass the email and privateKey that is used for authentication and just access the GCP KMS API directly.

For reference https://cloud.google.com/iam/docs/best-practices-for-using-and-managing-service-accounts#use-attached-service-accounts

The customer is running on CloudRun and sometimes Compute Engine on GCP so use attached service accounts. Their service accounts do not have user accessible keys on them.

The customer is using the MongoDB C#/.NET driver on Linux.

 During local development they use an environment variable GOOGLE_APPLICATION_CREDENTIALS that if you point it to a local service account file (purely for development purposes) that means the Google libraries behave exactly the same as when deployed in a attached service account environment.



 Comments   
Comment by Githook User [ 01/Mar/23 ]

Author:

{'name': 'Kevin Albertson', 'email': 'kevin.albertson@mongodb.com', 'username': 'kevinAlbs'}

Message: DRIVERS-2377 run apt-get update in setup-gce-instance.sh (#273)

run apt-get update in setup-gce-instance.sh
Branch: master
https://github.com/mongodb-labs/drivers-evergreen-tools/commit/0063d62bf3b1c0b827734c2d1236bba04d282a3b

Comment by Githook User [ 15/Feb/23 ]

Author:

{'name': 'Kevin Albertson', 'email': 'kevin.albertson@mongodb.com', 'username': 'kevinAlbs'}

Message: DRIVERS-2377 Define expansions for GCP earlier (#267)

  • define expansions for GCP earlier
Comment by Githook User [ 24/Jan/23 ]

Author:

{'name': 'Kevin Albertson', 'email': 'kevin.albertson@mongodb.com', 'username': 'kevinAlbs'}

Message: DRIVERS-2377 add GCPKMS_DISKSIZE option with default 20GB (#261)

add GCPKMS_DISKSIZE option with default 20GB
Branch: master
https://github.com/mongodb-labs/drivers-evergreen-tools/commit/b2c31c12d677a4f4ce506844d7b786db5c6bfeed

Comment by PM Bot [ 05/Dec/22 ]

Moved to Needs Triage because a linked PM issue PM-3084 was moved to Ready for Work.

Comment by Githook User [ 01/Dec/22 ]

Author:

{'name': 'Kevin Albertson', 'email': 'kevin.albertson@mongodb.com', 'username': 'kevinAlbs'}

Message: DRIVERS-2377 remove SSH keys on task completion (#251)

  • add ConnectTimeout=10
  • create VM with enable-oslogin

This enables deleting the SSH keys after the task completes
Branch: master
https://github.com/mongodb-labs/drivers-evergreen-tools/commit/c3bfc4c0150f7421c417b042b016d10bdf855fa5

Comment by Githook User [ 31/Aug/22 ]

Author:

{'name': 'Kevin Albertson', 'email': 'kevin.albertson@mongodb.com', 'username': 'kevinAlbs'}

Message: DRIVERS-2377 remove GCE_METADATA_HOST check (#1290)
Branch: master
https://github.com/mongodb/specifications/commit/eaec2671df55ba9d627e74bfc3f19a54ea6ac236

Comment by Githook User [ 10/Aug/22 ]

Author:

{'name': 'Kevin Albertson', 'email': 'kevin.albertson@mongodb.com', 'username': 'kevinAlbs'}

Message: DRIVERS-2377 Specify a non-root ssh user (#225)

  • capture output of last failed ssh attempt
Comment by Githook User [ 06/Aug/22 ]

Author:

{'name': 'Kevin Albertson', 'email': 'kevin.albertson@mongodb.com', 'username': 'kevinAlbs'}

Message: DRIVERS-2377 Support GCP attached service accounts when using GCP KMS (#1278)
Branch: master
https://github.com/mongodb/specifications/commit/847d9ba741201f9c9d1305831a9c60e8ab2a1544

Comment by Githook User [ 03/Aug/22 ]

Author:

{'name': 'Kevin Albertson', 'email': 'kevin.albertson@mongodb.com', 'username': 'kevinAlbs'}

Message: DRIVERS-2377 Add scripts for GCP KMS tests (#216)
Branch: master
https://github.com/mongodb-labs/drivers-evergreen-tools/commit/0e75b9ee5ac3952df51f782ff33de137933e89ed

Comment by David Stewart [ 26/Dec/21 ]

Hey Mongo Team,

As a Mongo DB Atlas customer, we would like to cast our vote for this as well. We are running in cloud run in a Node JS environment.

 

David 

Generated at Thu Feb 08 08:25:25 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.