Overview

Deliver Milestone 2 of the hermetic RBE container project: enable Windows and macOS builds to use Linux-hosted hermetic containers for cross-compilation and remote execution.

Parent Scope

• Epic: DEVPROD-33617
• Milestone: Windows/macOS containerized cross-compilation and RBE
• Estimate: 3 engineering weeks

Motivation

Windows and macOS currently depend on platform-specific build hosts and host-installed toolchains. Moving their build actions into pinned Linux hermetic containers will reduce differences between local, Evergreen, and RBE execution, and make more of the build graph eligible for RBE.

Scope of Work

• Implement and enable the macOS Linux-container cross-compilation path.
  • Route macOS compile, IDL, archive, debug-info, and link actions through the intended local-container/RBE split.
  • Keep macOS test execution on a native macOS host where required.
  • Support both arm64 and x86_64 macOS Evergreen variants.
• Implement and enable the Windows Linux-container cross-compilation path.
  • Package or provision the required Windows toolchain, SDK, sysroot, and runtime inputs for Linux-hosted execution.
  • Route Windows compile/link/IDL actions through RBE and run Windows tests using the supported host bridge.
  • Validate whether nested virtualization and AWS M8 instances are required for Docker on Windows.
• Define the supported container image, runtime, toolchain, and sysroot update paths.
• Ensure the cross-compilation action inputs are pinned, cacheable, and reproducible across local and remote execution.
• Add Evergreen coverage for the Windows and macOS variants and document operator/developer setup.

Acceptance Criteria

• macOS cross-compilation completes successfully from a supported macOS host using the pinned hermetic Linux image.
• Windows cross-compilation completes successfully from the supported Windows host configuration using the pinned hermetic Linux image.
• Compile, link, archive, IDL, and packaging actions use the intended local-container or RBE execution strategy for each platform.
• macOS and Windows test runners can execute the produced native-platform binaries and report results in Evergreen.
• The same action inputs and toolchain versions work for local and remote execution.
• Evergreen master/nightly coverage is enabled for the supported Windows and macOS variants.
• Container images and cross toolchains have documented update, signing, and rollback procedures.
• Failure messages clearly identify missing runtimes, toolchains, SDKs, sysroots, or unsupported host configurations.

Risks and Open Questions

• Windows may require nested virtualization and AWS M8 instances to support Docker; confirm whether this affects RDP, host provisioning, or task reliability.
• Determine how to distribute Windows system libraries and toolchain inputs through S3 or another controlled artifact source, including licensing and access requirements.
• Determine which tests must remain on native hosts and which can run inside Linux containers.
• Confirm the expected RBE capacity and cost impact of moving additional Windows/macOS actions to remote execution.
• Define the supported opt-out behavior when a local Docker-compatible runtime is unavailable.

Related Work

• SERVER-133625 tracks installing a Docker-compatible runtime on Evergreen macOS distro images.
• The proof of concept is described in the parent scope and was reported to validate macOS and Windows local containerized cross-compilation/RBE.

Non-Goal

This milestone does not attempt to make builds fully reproducible.