Actually apply the hardened HTTP retry policy when activating result task groups

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Fixed
    • Priority: Major - P3
    • 9.0.0-rc1
    • Affects Version/s: None
    • Component/s: Bazel
    • None
    • Fully Compatible
    • None
    • None
    • None
    • None
    • None
    • None
    • None

        1. Summary

      The `resmoke_tests` commit-queue task can fail in its post-test step even when all tests pass, because the result-task-group activation hits a transient Evergreen API error. Example: a `503 Service Unavailable` on `POST /rest/v2/versions/

      {version_id}

      /activate_tasks` failed an otherwise-green task.

      `buildscripts/evergreen_activate_result_tasks.py` already tried to harden against this by bumping the HTTP retry count, but the override was a *no-op*:

      ```python
      evg_api = RetryingEvergreenApi.get_api(...)
      evg_api._http_retry = Retry(total=DEFAULT_HTTP_RETRY_ATTEMPTS + 10, ...) # never takes effect
      ```

      `RetryingEvergreenApi.get_api()` builds a sticky `requests.Session` at construction time with an `HTTPAdapter(max_retries=...)` already mounted using the *default* `Retry`. The `session` property returns that sticky session and `_create_session()` is never called again, so reassigning `evg_api._http_retry` mutates an attribute nothing re-reads — the mounted adapter keeps the original 10-attempt policy.

        1. Change
      • Move the API construction into `get_evergreen_api()` which, after building the client, *re-mounts an `HTTPAdapter` with the new `Retry`* so the more aggressive policy is actually applied to outgoing requests.
      • Add `EXTRA_HTTP_RETRY_ATTEMPTS` constant to make the intent explicit.
      • Add `buildscripts/tests/test_evergreen_activate_result_tasks.py` plus the `py_library`/`py_test` bazel targets (the script had no test coverage before).
        1. Testing
      • New unit test asserts the adapter that actually serves requests uses `DEFAULT_HTTP_RETRY_ATTEMPTS + EXTRA_HTTP_RETRY_ATTEMPTS` retries. It is a genuine regression guard: I verified that the previous "reassign `_http_retry` only" form leaves the mounted adapter at the default 10 attempts (test fails), while the fix raises it to 20 (test passes).

      ```
      $ python -m unittest buildscripts.tests.test_evergreen_activate_result_tasks
      Ran 2 tests in 0.001s
      OK
      ```

        1. Notes
      • This does not address the root cause (Evergreen API returning 503s); it makes the client resilient to short outages. The underlying 503 should be reported to DevProd separately.
      • `SERVER-XXXX` placeholder in the title/commit — fill in before enqueueing.

            Assignee:
            Zack Winter
            Reporter:
            Zack Winter
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: