Retry URL lookup in debugsymb_mapper when artifacts are not yet visible

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • 9.1.0-rc0
    • Affects Version/s: None
    • Component/s: Bazel
    • None
    • v9.0, v8.3, v8.0, v7.0
    • 0
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Fixes BF-45377.

        1. Root cause

      The `generate_buildid_to_debug_symbols_mapping` task fails sporadically (~1/day across master and staging branches) with:

      ```
      ValueError: Debug symbols URL not found. URLs dict:

      {'Pip Requirements': ...}

      ```

      In the failing runs the compile task (`archive_dist_test`) succeeded and did attach its artifacts — e.g. for [this failure](https://spruce.mongodb.com/task/mongodb_mongo_v9.0_staging_linux_debug_aubsan_compile_required_generate_buildid_to_debug_symbols_mapping_11b3bf675f4f79d6743dd0aeba98103af5b99760_26_08_03_17_20_02?execution=0) the `Binaries` artifact exists on the compile task today, and the task's own `do setup` successfully downloaded the same tarball from S3 minutes earlier. But the artifact list returned by the Evergreen API to `debugsymb_mapper.py` contained only the artifacts attached early in the compile task (`Pip Requirements`) and was missing everything attached later (`Binaries`, `Binaries SHA256`, ...).

      This is because the Evergreen server now serves artifact reads from a secondary node ([DEVPROD-32223](https://jira.mongodb.org/browse/DEVPROD-32223), evergreen-ci/evergreen#10345), so artifacts attached by a recently-finished compile task can be invisible to the API for several minutes due to replication lag. The mapping task starts as soon as its dependency finishes, so it races with that lag.

        1. Fix

      `Mapper.setup_urls` in `buildscripts/debugsymb_mapper.py` now retries the multiversion URL lookup with exponential backoff (15s initial, capped at 120s, 8 attempts ≈ up to ~10 minutes) when either the `Binaries` URL or the debug symbols URL is missing, instead of immediately raising `ValueError`. The raise still happens after retries are exhausted, so genuinely missing artifacts (failed push, misconfigured variant) still fail the task.

      Also tightened the check: previously a missing `Binaries` URL on a non-SAN variant was not detected here and would fail later with a more obscure download error.

        1. Testing

      Added unit tests in `buildscripts/tests/test_debugsymb_mapper.py` covering:

      • success without retry
      • retry when the artifact list is initially incomplete, then succeeds
      • `ValueError` after retries are exhausted
      • SAN variant behavior (binaries URL doubles as symbols URL)

      `bazel test //buildscripts/tests:test_debugsymb_mapper` passes (11 tests).

            Assignee:
            Zack Winter
            Reporter:
            Zack Winter
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: