Automatically recover stale rootless Podman runtime state

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Fixed
    • Priority: Major - P3
    • 9.1.0-rc0
    • Affects Version/s: None
    • Component/s: Bazel
    • None
    • Fully Compatible
    • DevProd Test Infra 2026-08-25
    • 200
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Fixes BF-45694.

        1. Problem

      On hosts where the rootless Podman pause process has died, `podman info` fails with
      `invalid internal status, try resetting the pause process with "podman system migrate"`.
      The existing recovery path in `_recover_stale_podman_runtime` detected this state but
      deliberately never ran the migration, so `buildscripts_test` (and any hermetic-container
      Bazel invocation) failed on every retry until someone manually repaired the host.

        1. Changes

      `bazel/wrapper_hook/hermetic_container_integration.py` — once the stale state is confirmed
      under the recovery lock, `_recover_stale_podman_runtime` now runs `podman system migrate`
      and re-verifies with `podman info`, reporting a specific failure detail if either step fails.
      Adds `_podman_migration_is_allowed` and `_podman_has_running_containers` helpers.

      `buildscripts/podman_recovery.py` — the standalone wrapper gets the same logic
      (`_migration_is_allowed`, `_has_running_containers`) so both entry points behave identically.

        1. Safety
      • Migration only runs when `podman ps --quiet` shows no running containers, so it never
        stops containers a user still cares about. If containers are running, the previous
        manual-recovery message is printed and the command fails as before.
      • A `podman ps` that fails with the same stale-status error is treated as "no reachable
        containers": the pause process is dead, so those containers are already unusable. This is
        the CI case.
      • `MONGO_BAZEL_PODMAN_AUTO_MIGRATE=0` opts out entirely and restores the previous behavior.
      • Recovery still happens under the existing per-user exclusive lock, and still re-checks
        runtime health after acquiring it, so concurrent Bazel invocations do not migrate twice.
      • Docker and non-stale Podman failures are untouched.
        1. Testing

      Unit tests updated in `buildscripts/tests/test_podman_recovery.py` and
      `bazel/wrapper_hook/hermetic_container_integration_test.py` covering: migrate-when-idle,
      migrate-when-containers-unreachable, refuse-when-containers-running, migrate-failure
      reporting, and the env-var opt-out.

      • `python -m unittest buildscripts.tests.test_podman_recovery` — 10 passed
      • `python -m unittest bazel.wrapper_hook.hermetic_container_integration_test` — 193 passed

            Assignee:
            Daniel Moody
            Reporter:
            Steve McClure
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: