Fixes BF-45694.
-
- Problem
On hosts where the rootless Podman pause process has died, `podman info` fails with
`invalid internal status, try resetting the pause process with "podman system migrate"`.
The existing recovery path in `_recover_stale_podman_runtime` detected this state but
deliberately never ran the migration, so `buildscripts_test` (and any hermetic-container
Bazel invocation) failed on every retry until someone manually repaired the host.
-
- Changes
`bazel/wrapper_hook/hermetic_container_integration.py` — once the stale state is confirmed
under the recovery lock, `_recover_stale_podman_runtime` now runs `podman system migrate`
and re-verifies with `podman info`, reporting a specific failure detail if either step fails.
Adds `_podman_migration_is_allowed` and `_podman_has_running_containers` helpers.
`buildscripts/podman_recovery.py` — the standalone wrapper gets the same logic
(`_migration_is_allowed`, `_has_running_containers`) so both entry points behave identically.
-
- Safety
- Migration only runs when `podman ps --quiet` shows no running containers, so it never
stops containers a user still cares about. If containers are running, the previous
manual-recovery message is printed and the command fails as before. - A `podman ps` that fails with the same stale-status error is treated as "no reachable
containers": the pause process is dead, so those containers are already unusable. This is
the CI case. - `MONGO_BAZEL_PODMAN_AUTO_MIGRATE=0` opts out entirely and restores the previous behavior.
- Recovery still happens under the existing per-user exclusive lock, and still re-checks
runtime health after acquiring it, so concurrent Bazel invocations do not migrate twice. - Docker and non-stale Podman failures are untouched.
-
- Testing
Unit tests updated in `buildscripts/tests/test_podman_recovery.py` and
`bazel/wrapper_hook/hermetic_container_integration_test.py` covering: migrate-when-idle,
migrate-when-containers-unreachable, refuse-when-containers-running, migrate-failure
reporting, and the env-var opt-out.
- `python -m unittest buildscripts.tests.test_podman_recovery` — 10 passed
- `python -m unittest bazel.wrapper_hook.hermetic_container_integration_test` — 193 passed