[CSHARP-4395] C# Driver caches AWS credentials assumed for IAM Role until the first error Created: 02/Nov/22  Updated: 28/Oct/23  Resolved: 08/Nov/22

Status: Closed
Project: C# Driver
Component/s: None
Affects Version/s: None
Fix Version/s: 2.19.0

Type: Bug Priority: Unknown
Reporter: Andrii Litvinov Assignee: Dmitry Lukyanov (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Issue split
split from DRIVERS-2493 Ensure Auth Environment Variables are... Implementing
Related
related to CSHARP-3740 Add native support for AWS IAM Roles ... Closed
related to CSHARP-4273 Cache AWS Credentials Where Possible Closed
Upstream Changes Summary:

DRIVERS-2493:

Summary of required changes

  • Ensure that AWS credentials fetched from environment variables are handled appropriately.
  • Add tests to verify environment variable handling

Additional background

Please see https://github.com/mongodb/specifications/commit/875446db44aade414011731840831f38a6c668dffor the specification change.

Please see https://github.com/mongodb/mongo-python-driver/commit/ff94b0e3094f6bf08645ff0a491ec9b51f504b53 for a reference implementation in Python.

Integration test

Drivers are expected to add integration tests as described in the specification change


 Description   

Summary

Our services deployed to EKS and connect to MongoDB Atlas. We use IAM role authentication, which is not yet natively supported by the driver.

We assume role to get credentials and set corresponding environment variables for MongoDB Driver to pick up:

using var stsClient = new AmazonSecurityTokenServiceClient();
 
var assumeRoleResponse = await stsClient.AssumeRoleWithWebIdentityAsync(new AssumeRoleWithWebIdentityRequest
{
    DurationSeconds = (int) CredentialsLifetime.TotalSeconds,
    RoleArn = roleArn,
    WebIdentityToken = GetWebIdentityToken(),
    RoleSessionName = "default"
});
 
var awsCredentials = assumeRoleResponse.Credentials;
 
Environment.SetEnvironmentVariable("AWS_ACCESS_KEY_ID", awsCredentials.AccessKeyId);
Environment.SetEnvironmentVariable("AWS_SECRET_ACCESS_KEY", awsCredentials.SecretAccessKey);
Environment.SetEnvironmentVariable("AWS_SESSION_TOKEN", awsCredentials.SessionToken);

Few minutes before the expiration we refresh credentials and reset environment variables. Up until v2.18.0 everything was working fine. New version has introduced credentials caching. This new caching assumes that credentials from environment variables never expire, which results in exception each time the token expires.

As a workaround we have to manually clear the credentials cache upon refres, which is quite hacky:

private static Action CreateClearAction()
{
    var authenticators = typeof(MongoAWSAuthenticator).Assembly
        .GetType("MongoDB.Driver.Core.Authentication.External.ExternalCredentialsAuthenticators")!
        .GetProperty("Instance", BindingFlags.Public | BindingFlags.Static)!
        .GetValue(null);
 
    var cache = authenticators!
        .GetType()
        .GetProperty("Aws")!
        .GetValue(authenticators);
 
    var clear = cache!.GetType().GetMethod("Clear");
 
    return (Action)Delegate.CreateDelegate(typeof(Action), cache, clear!);
}

It would be great to have some built-in support for cache invalidation until the driver receives native support for IAM roles. As an idea, the driver could check the cached value of the environment variables and the current value to decide whether the cache has expired.

Motivation

Who is the affected end user?

Users that use IAM Roles authentication in EKS.

How does this affect the end user?

User has to retry their operation.

How likely is it that this problem or use case will occur?

Easily reproducible.

If the problem does occur, what are the consequences and how severe are they?

The problem causes transient errors to our end users.

Is this issue urgent?

Not urgent as we implemented a workaround.

Is this ticket required by a downstream team?

No.

Is this ticket only for tests?

No.



 Comments   
Comment by Githook User [ 08/Nov/22 ]

Author:

{'name': 'Dmitry Lukyanov', 'email': 'dmitry.lukyanov@mongodb.com', 'username': 'DmitryLukyanov'}

Message: CSHARP-4395: C# Driver caches AWS credentials assumend for IAM Role untill the first error. (#933)
Branch: master
https://github.com/mongodb/mongo-csharp-driver/commit/1f761191fb44d0c53dcbfe5d739a20f057be6c7e

Comment by James Kovacs [ 02/Nov/22 ]

Hi, andrii.litvinov@gmail.com,

Thank you for reporting this issue in the .NET/C# Driver. The intent of AWS credential caching is to avoid hitting rate limits on AWS credential endpoints and improving connection performance by using cached credentials if available to avoid an additional network roundtrip to fetch credentials. If you are using environment variables, there is no reason nor need to cache the credentials as they're trivially available on demand.

We are clarifying the Auth spec to only cache AWS credentials fetched from a network endpoint, but not from environment variables. (See DRIVERS-2493.) We will make this change in an upcoming release of the .NET/C# Driver (along with other affected drivers). In the meantime your workaround seems reasonable (if a bit hacky) until we can get this issue fixed in the driver itself.

FYI - We are implementing native support for EKS in CSHARP-3740, which will also be available in an upcoming release.

Thank you again for clearly explaining the issue and providing supporting code to demonstrate the problem/workaround. Much appreciated!

Sincerely,
James

Generated at Wed Feb 07 21:48:06 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.