[DRIVERS-2411] Support the Azure VM-assigned Managed Identity for Automatic KMS Credentials Created: 10/Aug/22  Updated: 29/Aug/23  Resolved: 14/Jul/23

Status: Closed
Project: Drivers
Component/s: Client Side Encryption
Fix Version/s: None

Type: Improvement Priority: Unknown
Reporter: Colby Pike Assignee: Colby Pike
Resolution: Done Votes: 0
Labels: RDY
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by RUBY-3088 Support the Azure VM-assigned Managed... Closed
Duplicate
is duplicated by DRIVERS-1780 Support for Azure Managed Identities Closed
is duplicated by DRIVERS-2457 Define Additional Prose Tests for Aut... Closed
Gantt End to End
has to be finished together with DRIVERS-2600 Clean up Azure resources on task failure Closed
Issue split
split to JAVA-4706 Support the Azure VM-assigned managed... Closed
split to CDRIVER-4454 Support the Azure VM-assigned Managed... Closed
split to CSHARP-4294 Support the Azure VM-assigned Managed... Closed
split to CXX-2565 Support the Azure VM-assigned Managed... Closed
split to GODRIVER-2521 Support the Azure VM-assigned Managed... Closed
split to MOTOR-1017 Support the Azure VM-assigned Managed... Closed
split to NODE-4537 Support the Azure VM-assigned Managed... Closed
split to PHPLIB-938 Support the Azure VM-assigned Managed... Closed
split to PYTHON-3396 Support the Azure VM-assigned Managed... Closed
split to RUBY-3088 Support the Azure VM-assigned Managed... Closed
split to RUST-1442 Support the Azure VM-assigned Managed... Closed
Problem/Incident
Related
related to DRIVERS-2457 Define Additional Prose Tests for Aut... Closed
related to MONGOCRYPT-473 Recognize Azure Automatic KMS Requests Closed
is related to DRIVERS-2464 Consider clear azure cached token in ... Backlog
Driver Changes: Needed
Quarter: FY23Q3, FY24Q1
Downstream Changes Summary:

Implementation

libmongocrypt 1.6.0 or higher is required. Binaries for 1.6.0 are available on the upload-all task.

The spec changes introduce another method of obtaining KMS credentials automatically, much like with GCP and AWS:

  • When kmsProviders contains an empty azure property, it indicates a request for automatic Azure credentials.
  • To obtain credentials, issue an HTTP request to the Azure Instance Metadata Service (IMDS).
  • IMDS will issue an accessToken that can be used to query the Azure Key Vault (if the instance has sufficient permissions).
  • Additionally, this version of auto-KMS credentials institutes a token caching requirement.

The associated spec changes are specified here: https://github.com/mongodb/specifications/commit/d6b8cce6abb3b8e1a0b8f1dc7ee737e18322cfce

The initial implementation for the C driver is here: https://github.com/mongodb/mongo-c-driver/commit/686bff81f565f93db83d99902ce1c3a6f89922c7

Mock server tests

Mock server tests specified here:
https://github.com/mongodb/specifications/commit/e780e91d708fe9c004a0b0023387baa850282881

The mock server is available here: https://github.com/mongodb-labs/drivers-evergreen-tools/blob/master/.evergreen/csfle/fake_azure.py

Please see https://github.com/mongodb/mongo-c-driver/commit/671a15154f0dd0e4af3c8df2ac08dfe4acf01795#diff-d353a218f6d4ac77dfb35cc757a96af121a9ce1d3cf7b01535fa23e6d0c58016R98 for a reference implementation of the mock server tests in C.

Integration tests

Integration tests are specified here:
https://github.com/mongodb/specifications/commit/cf778cb8add04c0c6d8f366e6352f3d0ac9c1694

Scripts in the drivers-evergreen-tools .evergreen/csfle/azurekms directory may be used to create the temporary Azure Virtual Machine. Get credentials from DRIVERS-2411 Test Credentials.

To test, add an Evergreen task group to do the following:

  • Create an Azure VM instance in a setup_group.
  • Destroy the Azure VM instance in a teardown_group. Using a teardown_group will destroy the instance if the task fails.

Add a task in the task group to do the following:

  • Build and copy files to the remote Azure VM.
  • Install necessary dependencies on the remote Azure VM instance.
  • Run the test remotely.

Please see https://github.com/mongodb/mongo-c-driver/pull/1124 and https://github.com/mongodb/mongo-c-driver/pull/1234/ for a reference implementation of the integration tests in C.

It may be helpful to refer to driver tests for MONGODB-AWS ECS. The ECS tests perform a similar flow (copying and running a test on a remote ECS instance).

Driver Compliance:
Key Status/Resolution FixVersion
CDRIVER-4454 Fixed 1.24.0
CXX-2565 Works as Designed 3.8.0
CSHARP-4294 Fixed 2.19.0
GODRIVER-2521 Fixed 1.12.0
JAVA-4706 Fixed 4.8.0
NODE-4537 Fixed 5.2.0, mongodb-client-encryption-2.7.1
MOTOR-1017 Duplicate
PYTHON-3396 Fixed pymongocrypt-1.4, 4.3.3
PHPLIB-938 Done
RUBY-3088 Fixed 2.19.0
RUST-1442 Fixed 2.6.0
SWIFT-1625 Won't Do

 Description   

Summary

At present, using Azure Key Vault for KMS requires a tenant ID, client ID, and client secret (password) in order to obtain an OAuth2 token to subsequently query the Azure Key Vault for key management.

Azure VMs are automatically assigned a Managed Identity which allows the VM to obtain an OAuth2 token by querying a private metadata HTTP endpoint without any special credentials. The token obtained can then be used to request access to Azure resources, including the Key Vault, if applicable.

We have added support for automatic KMS credentials for AWS (DRIVERS-2280) and GCP (DRIVERS-2377). The Azure VM Managed Identity can be used for the same purpose, removing the need for the client to provide credentials if they are running within an Azure VM that has the appropriate access to the Key Vault:

  1. Allow the user to omit Azure credentials as part of kmsProviders, and request that credentials be obtained automatically from the environment.
  2. Attempt to load OAuth2 token from the Azure metadata endpoint. If we are running within an Azure VM, this token corresponds to the managed identity of that VM.
  3. Use the token from (2) as the KMS credential for client encryption.

Motivation

Who is the affected end user?

All end users of client-side encryption that wish to use Azure Key Vault as their KMS provider, and are running their client within an Azure VM.

How does this affect the end user?

Supporting the VM's managed identity alleviates the need to manage a separate set of credentials for their client application.

How likely is it that this problem or use case will occur?

As Azure KMS users are likely running within an Azure VM, most Azure users will likely find benefit in delegating the credential management to the Azure platform.

If the problem does occur, what are the consequences and how severe are they?

Managing and securing additional credentials creates an additional security concern and barrier to adoption for client-side encryption, whereas requesting the client driver to automatically work with the Azure host is simpler, more secure, and less error-prone.

Is this issue urgent?

The first mention of using Azure Managed Identities appears in December of 2020. The recent addition of automatic credentials for AWS and GCP makes this change prudent to match platform support.

Is this ticket required by a downstream team?

No

Is this ticket only for tests?

No



 Comments   
Comment by Githook User [ 15/Apr/23 ]

Author:

{'name': 'Kevin Albertson', 'email': 'kevin.albertson@mongodb.com', 'username': 'kevinAlbs'}

Message: DRIVERS-2411 remove install-az.sh (#293)

  • note which distros have `az` installed
  • remove reference to `install-az.sh`
  • check for az version >= 2.25.0
  • add link to error message
Comment by Githook User [ 24/Feb/23 ]

Author:

{'name': 'Kevin Albertson', 'email': 'kevin.albertson@mongodb.com', 'username': 'kevinAlbs'}

Message: DRIVERS-2411 Do not skip install of `azure-cli` if `az` is detected (#271)

Comment by Githook User [ 06/Nov/22 ]

Author:

{'name': 'Kevin Albertson', 'email': 'kevin.albertson@mongodb.com', 'username': 'kevinAlbs'}

Message: DRIVERS-2411 permit global cache (#1340)
Branch: master
https://github.com/mongodb/specifications/commit/c796a746327af27f9070bd1791ced6bce9176453

Comment by Githook User [ 03/Nov/22 ]

Author:

{'name': 'Kevin Albertson', 'email': 'kevin.albertson@mongodb.com', 'username': 'kevinAlbs'}

Message: DRIVERS-2411 fix handling of `AZUREKMS_IMAGE` (#243)

  • fix handling of `AZUREKMS_IMAGE`
  • pin to the latest version of Debian 11
Comment by Githook User [ 21/Oct/22 ]

Author:

{'name': 'Kevin Albertson', 'email': 'kevin.albertson@mongodb.com', 'username': 'kevinAlbs'}

Message: DRIVERS-2411 add integration tests for Azure KMS (#1325)

DRIVERS-2411 add integration tests
Branch: master
https://github.com/mongodb/specifications/commit/cf778cb8add04c0c6d8f366e6352f3d0ac9c1694

Comment by Githook User [ 20/Oct/22 ]

Author:

{'name': 'Kevin Albertson', 'email': 'kevin.albertson@mongodb.com', 'username': 'kevinAlbs'}

Message: DRIVERS-2411 add test scripts for Azure VM (#239)

Add scripts for integration testing.
Branch: master
https://github.com/mongodb-labs/drivers-evergreen-tools/commit/976d3c8020e95040e5d413b345f50c106b7bc3b7

Comment by Githook User [ 17/Oct/22 ]

Author:

{'name': 'vector-of-bool', 'email': 'vectorofbool@gmail.com', 'username': 'vector-of-bool'}

Message: DRIVERS-2411 Prose Tests for IMDS Communication (#1314)

Comment by Githook User [ 10/Oct/22 ]

Author:

{'name': 'vector-of-bool', 'email': 'vectorofbool@gmail.com', 'username': 'vector-of-bool'}

Message: DRIVERS-2411 Add a fake_azure server for testing (#235)

Add a fake_azure server for testing
Branch: master
https://github.com/mongodb-labs/drivers-evergreen-tools/commit/651efa00f0118fff65e075706a80f4fafd7b0b04

Comment by Githook User [ 26/Sep/22 ]

Author:

{'name': 'vector-of-bool', 'email': 'vectorofbool@gmail.com', 'username': 'vector-of-bool'}

Message: DRIVERS-2411 Add behavior for automatic Azure KMS credentials (#1291)

Comment by Githook User [ 09/Sep/22 ]

Author:

{'name': 'vector-of-bool', 'email': 'vectorofbool@gmail.com', 'username': 'vector-of-bool'}

Message: CDRIVER-4454 Automatic Azure KMS Credentials (#1097)

Generated at Thu Feb 08 08:25:30 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.