Summary

Our services deployed to EKS and connect to MongoDB Atlas. We use IAM role authentication, which is not yet natively supported by the driver.

We assume role to get credentials and set corresponding environment variables for MongoDB Driver to pick up:

using var stsClient = new AmazonSecurityTokenServiceClient();

var assumeRoleResponse = await stsClient.AssumeRoleWithWebIdentityAsync(new AssumeRoleWithWebIdentityRequest
{
    DurationSeconds = (int) CredentialsLifetime.TotalSeconds,
    RoleArn = roleArn,
    WebIdentityToken = GetWebIdentityToken(),
    RoleSessionName = "default"
});

var awsCredentials = assumeRoleResponse.Credentials;

Environment.SetEnvironmentVariable("AWS_ACCESS_KEY_ID", awsCredentials.AccessKeyId);
Environment.SetEnvironmentVariable("AWS_SECRET_ACCESS_KEY", awsCredentials.SecretAccessKey);
Environment.SetEnvironmentVariable("AWS_SESSION_TOKEN", awsCredentials.SessionToken);

Few minutes before the expiration we refresh credentials and reset environment variables. Up until v2.18.0 everything was working fine. New version has introduced credentials caching. This new caching assumes that credentials from environment variables never expire, which results in exception each time the token expires.

As a workaround we have to manually clear the credentials cache upon refres, which is quite hacky:

private static Action CreateClearAction()
{
    var authenticators = typeof(MongoAWSAuthenticator).Assembly
        .GetType("MongoDB.Driver.Core.Authentication.External.ExternalCredentialsAuthenticators")!
        .GetProperty("Instance", BindingFlags.Public | BindingFlags.Static)!
        .GetValue(null);

    var cache = authenticators!
        .GetType()
        .GetProperty("Aws")!
        .GetValue(authenticators);

    var clear = cache!.GetType().GetMethod("Clear");

    return (Action)Delegate.CreateDelegate(typeof(Action), cache, clear!);
}

It would be great to have some built-in support for cache invalidation until the driver receives native support for IAM roles. As an idea, the driver could check the cached value of the environment variables and the current value to decide whether the cache has expired.

Motivation

Who is the affected end user?

Users that use IAM Roles authentication in EKS.

How does this affect the end user?

User has to retry their operation.

How likely is it that this problem or use case will occur?

Easily reproducible.

If the problem does occur, what are the consequences and how severe are they?

The problem causes transient errors to our end users.

Is this issue urgent?

Not urgent as we implemented a workaround.

Is this ticket required by a downstream team?

No.

Is this ticket only for tests?

No.