Problem

The antithesis image build and push Evergreen function generates GitHub App tokens for 10gen/QA and 10gen/jstestfuzz up front (steps 4.1/4.2 in etc/evergreen_yml_components/definitions.yml), writes them into expansions, and then invokes the long-running image build (step 4.6). resmoke consumes those tokens much later, in DockerClusterImageBuilder._clone_qa_repo_to_build_context / _clone_jstestfuzz_to_build_context (buildscripts/resmokelib/testing/docker_cluster_image_builder.py lines 508 and 528), via _clone_repo at line 636.

GitHub App installation tokens expire after 1 hour. Any antithesis image build that takes longer than an hour to reach the QA clone fails authentication, even though the token was created successfully.

Evidence

Observed in BF-45086. Example task: antithesis_sharding_basic_js_commands on enterprise-ubuntu2204-64-libvoidstar, mongodb-mongo-v9.0-staging @ 96e71d2f.

Token creation succeeded:

Requesting a GitHub dynamic access token with owner:10gen, repository:QA, permissions:[Contents:read, Metadata:read]
Created a GitHub dynamic access token. The token has the following permissions: [Contents:read, Metadata:read]

Step 4.6 then ran for 1h26m10s, and the QA clone at the very end of it failed:

Cloning QA repo to build context...
Found token for 10gen/QA git repo, using http clone
...
  cmdline: git clone -v -- https://*****:*****@github.com/10gen/QA.git buildscripts/antithesis/base_images/workload/QA
  stderr: 'Cloning into 'buildscripts/antithesis/base_images/workload/QA'...
remote: Invalid username or token. Password authentication is not supported for Git operations.
fatal: Authentication failed for 'https://github.com/10gen/QA.git/'

The token was roughly 86 minutes old at clone time. This is not a missing project variable and not a permissions problem – the token had the right scopes, it had simply expired.

Contributing factor (tracked separately)

The build was pushed past the 1-hour window by a concurrent S3 IAM regression producing long retry chains on every artifact download:

api error AccessDenied: User: arn:aws:iam::557821124784:user/srv-evg-mongodb-mongo-s3-key
is not authorized to perform: s3:ListBucket on resource: "arn:aws:s3:::mciuploads"
because no identity-based policy allows the s3:ListBucket action

Restoring that IAM policy will hide this bug again, but the token-lifetime fragility remains: any antithesis image build exceeding 60 minutes will fail the same way.

Suggested fix

Either:

• clone 10gen/QA and 10gen/jstestfuzz in a dedicated Evergreen step immediately after github.generate_token, before the multi-hour docker build begins; or
• have _clone_repo obtain a fresh token at clone time rather than reading a pre-computed expansion.

The first option is the smaller change and keeps token handling in the Evergreen config.