Retry on MaxTimeMSExpired in blockUntilIndexQueryable

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Fixed
    • Priority: Major - P3
    • 9.1.0-rc0
    • Affects Version/s: None
    • Component/s: Build
    • None
    • Query Integration
    • Fully Compatible
    • QI 2025-02-17
    • 0
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Driving build failure: [BF-45732](https://jira.mongodb.org/browse/BF-45732) — `failed: with_mongot_extension_fully_enabled_sharded_cluster on amazon2023-arm64-mongot-integration-extensions-secure [mongodb-mongo-v9.0-staging @ 556d280e] (foo.js)`

        1. What this PR does

      `blockUntilIndexQueryable()` in `src/mongo/db/query/search/search_index_command_testing_helper.h` issues each `$listSearchIndexes` round trip with a 60s `kRemoteCommandTimeout`, inside a loop whose intended budget is the 10-minute `maxTimeout`. When a round trip exceeded 60s, the target mongod killed the command and replied with a command-level `MaxTimeMSExpired`. That code is not in the `RetriableError` category, and it was not one of the two statuses the loop special-cases, so it fell through to `uassertStatusOKWithContext(cmdStatus, "blockUntilIndexQueryable failed")` and aborted on the *first* attempt — the 10-minute budget was never reached.

      This PR:

      • Retries on command-level `MaxTimeMSExpired` alongside the existing `SearchIndexManagementHostUnreachable` case, so a single slow round trip is absorbed by the retry budget rather than failing the helper outright.
      • Threads a `lastError` through both retry paths and into the `9638406` "not replicated and queryable within the max timeout" assertion, so exhausting the budget names the underlying cause instead of reporting an opaque timeout.
        1. Motivation

      Without the second half of this change, retrying alone would paper over a genuinely hung or crashed mongot for a full 10 minutes and then report a bare timeout. Carrying the last error into the assertion keeps that case diagnosable — relevant because sibling [BF-45535](https://jira.mongodb.org/browse/BF-45535) shows the same surface error text (`blockUntilIndexQueryable failed` / `MaxTimeMSExpired`) but is attributed to a mongot JVM crash, i.e. a different root cause that terminates at this same assertion.

      This continues the retry-classification work of SERVER-127065 (`CommandFailed` / mongot transiently unreachable) and SERVER-127787 (`NetworkInterfaceExceededTimeLimit`). Both are already present on master and v9.0-staging; neither covered `MaxTimeMSExpired`.

            Assignee:
            Finley Lau
            Reporter:
            Finley Lau
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: