Retry transient Evergreen 401/404 with a capped budget

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Fixed
    • Priority: Major - P3
    • 9.1.0-rc0
    • Affects Version/s: None
    • Component/s: Build
    • None
    • Fully Compatible
    • DevProd Test Infra 2026-08-11
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Task generation fails when the Evergreen API returns a transient 401 or 404. Neither is in `evergreen.py`'s default retry list, so the error propagates and kills the task. These do happen spuriously — we have seen a 401 on the second page of a paginated `tasks_by_build` call whose first page succeeded on the same session with the same credentials.

      Retry them, but on a smaller budget than the 5xx codes. Unlike those, a 401 or 404 is usually genuine, so spending the full ~21 minute budget before reporting it is wasteful. `TransientAuthRetry` allows 3 attempts spaced a flat 120s apart, a ~6 minute ceiling; 5xx and 429 keep the full budget.

      Behavior on exhaustion is unchanged: callers still get the same `HTTPError`, just later.

      Tests drive real retry chains rather than hand-built state, and cover the cap, the spacing, and that the 5xx budget is unaffected.

      One limitation worth knowing: the 3 attempts are a budget for the whole call, not per page, so a paginated fetch that blips on several different pages shares them.

            Assignee:
            Teodor Voinea
            Reporter:
            Teodor Voinea
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: