Goal

The VOYAGEAI_API_KEY in the drivers/ai-ml-pipeline-testing AWS Secrets Manager vault has expired, failing the Voyage-dependent tasks in the AI Frameworks nightly CI. Provision a replacement key, store it in the vault, and make the Voyage endpoint configurable so the suite can run against an Atlas AI Models key (billed to the project) rather than a direct voyageai.com platform key.

Blocked by

Nothing. All work is in repos we own.

Current state

Verified on origin/main at commit 7fd88e0 (2026-08-03).

Item	Status
Secret name in vault	VOYAGEAI_API_KEY in drivers/ai-ml-pipeline-testing, fetched by .evergreen/fetch-secrets.sh via setup-secrets.sh in drivers-evergreen-tools
Key type currently in use	Direct VoyageAI platform key (pa- prefix), so it authenticates api.voyageai.com only. Not billed to the Atlas project.
Key expired	Cause of the nightly failures
Endpoint configurability	.evergreen/mongodb-community-search/mongot.conf line 26 hardcodes providerEndpoint to the api.voyageai.com embeddings URL. No override mechanism anywhere.
Atlas AI Models key provisioned	Never done for this project
Branch rotate-voyage_key	Exists locally but is identical to origin/main, so no work has started

Consumers of the key on main (5 files)

Affected CI

Build variant test-pymongo-voyageai-python-rhel (.evergreen/config.yml line 488, DIR: pymongo-voyageai) running tasks test-pymongo-voyageai-local (line 246) and test-pymongo-voyageai-remote (line 253). Also any variant using the setup community atlas function, which drives mongot auto-embedding through mongot.conf.

Pitfalls

1. Vault write access. CONTRIBUTING.md links the vault access wiki page but only describes reading secrets via fetch-secrets.sh. Rotating one requires write access to drivers/ai-ml-pipeline-testing, which is a separate grant.
2. The two key types are mutually exclusive. An Atlas AI Models key authenticates ai.mongodb.com only; a voyageai.com platform key authenticates api.voyageai.com only. Swapping the key value alone will fail with 401. Key and endpoint must change together, as a matched pair. Nothing in the repo or the process docs states this.
3. mongot endpoint compatibility is unverified. The providerEndpoint in mongot.conf expects a Voyage-shaped /v1/embeddings API. Whether mongot can talk to ai.mongodb.com at all (path shape, auth header, supported models) has not been established, and should be checked before committing to Atlas keys for the community-search path. This may force a split: Atlas key for the Python suite, platform key for mongot.
4. Downstream repo has no endpoint plumbing. In mongodb-labs/pymongo-voyageai-multimodal, pymongo_voyageai_multimodal/client.py line 175 reads self._vo = voyageai_client or Client(api_key=voyageai_api_key). No base-URL argument is threaded through. The constructor does accept a prebuilt voyageai_client, so injecting a configured client is a viable escape hatch that avoids an upstream API change.

Steps

1. Confirm vault write access to drivers/ai-ml-pipeline-testing, and request it if absent.
2. Decide the key strategy. See Decisions needed below.
3. Establish whether the voyageai Python SDK supports a base-URL override (constructor argument, module-level api_base, or environment variable) on the version pinned by pymongo-voyageai-multimodal. This determines whether step 7 is a config change or a code change.
4. Verify whether mongot accepts an ai.mongodb.com providerEndpoint, or confirm that it cannot.
5. Provision the new key or keys: Atlas > AI Models for the project-billed key, and/or the voyageai.com platform console.
6. Add to the vault. If both key types are needed, introduce a second secret rather than overloading one name. Suggest keeping VOYAGEAI_API_KEY for the platform key, adding VOYAGEAI_ATLAS_API_KEY, and adding VOYAGEAI_API_BASE for the endpoint.
7. Thread the endpoint through the consumers: export it in provision-atlas.sh and setup-remote.sh alongside the key; template providerEndpoint in mongot.conf from start-services.sh instead of hardcoding it; pass the base URL into the client in pymongo-voyageai/run.sh, or upstream per step 3.
8. Run the affected variant as an Evergreen patch build and confirm it is green.
9. Record the new key's expiry date and set a rotation reminder, so this recurs as a scheduled task rather than a CI outage.

Acceptance criteria

• test-pymongo-voyageai-local and test-pymongo-voyageai-remote pass in a patch build and in the following nightly.
• Voyage-dependent community-search tasks pass, or are explicitly documented as pinned to a platform key with the reason recorded.
• No Voyage endpoint is hardcoded in the repo. git grep api.voyageai.com returns only defaults that a variable can override.
• Voyage embedding usage appears on the Atlas project bill for whichever suites use the Atlas key.
• CONTRIBUTING.md states which key type each suite requires, where it is provisioned, and how to rotate it.
• The key's expiry date is recorded somewhere that surfaces before it lapses.

Decisions needed

1. Key strategy. The two endpoints do not interoperate and do not accept each other's keys, so "make either key work" means making endpoint and key a configurable pair, not auto-detection. Options: (a) Atlas key everywhere, which is best for billing but blocked if mongot cannot reach ai.mongodb.com; (b) Atlas key for the Python suite and platform key for mongot community-search, which is pragmatic but leaves two secrets to rotate; (c) platform key everywhere with endpoint plumbing landed for a later migration, which is the fastest path to green CI but defers the billing win.
2. Restore CI first, or migrate first? Options: (a) drop in a like-for-like platform key now to unbreak the nightly, then migrate to Atlas keys as follow-up work; (b) do the migration in one pass and leave CI red for longer. Option (a) is recommended, since it decouples the outage from the design question.
3. Secret naming. Reusing VOYAGEAI_API_KEY for an Atlas key silently changes its meaning for anyone reading secrets-export.sh. Options: (a) add a distinctly named secret and keep the old name meaning "platform key"; (b) redefine the existing name and update all five consumers plus the docs.