ExportXMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fixed
    • Priority: Major - P3
    • 9.1.0-rc0, 9.1.0-rc1025
    • Affects Version/s: None
    • Component/s: None
    • None
    • Storage Engines - Server Integration
    • Fully Compatible
    • ALL
    • SESIWhinery 2026-11-17
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Problem

      When the macOS build variants move from the macos-14 to the macos-15 distros (DEVPROD-45657 / SERVER-137597), the queryableWT task fails on both enterprise macOS variants. 3 of the 5 tests in src/mongo/db/modules/enterprise/jstests/queryable_wt/ fail: simple_rs_restore.js, simple_rs_pit_restore.js (arm64) and simple_standalone_restore.js (amd64). The task is green on macos-14 in mainline the same day (0f094cb6, Python 3.13.15 on both fleets).

      Evidence: https://spruce.corp.mongodb.com/version/6ad903bc0a71f20007041f5a

      • enterprise-macos-arm64 / queryableWT (host mac-m4-1b-1, macos-15-arm64-gui)
      • enterprise-macos / queryableWT (host mac1-1a-1, macos-15-amd64-gui)

      Symptom

      The restored mongod (--storageEngine queryable_wt --queryableBackupApiUri localhost:8080) cannot reach the resmoke-side queryable HTTP server:

      "ctx":"CurlConnPool-0","msg":"Connecting","attr":{"hostAndPort":"localhost:8080"}
      "msg":"Ending connection due to bad connection status","attr":{"hostAndPort":"localhost:8080","error":"OperationFailed: Bad HTTP response from API server: Timeout was reached"}
      ... (repeats for ~65s) ...
      "c":"WT","msg":"WiredTiger error message","attr":{"message":"wiredtiger_open: [WT_VERB_EXTENSION][ERROR]: ListDir: Bad HTTP response from the ApiServer: Bad HTTP response from API server: Timeout was reached"}
      mongo program was not running at 20041, process ended with exit code: 1
      

      In the arm64 run the first test (simple_standalone_restore.js) passes and the server access log shows ~1000 successful /os_list / /os_read / /os_wt_recovery_write requests. QueryableServerHook then stops the server thread after the test and starts a fresh one before the next test (buildscripts/resmokelib/testing/hooks/queryable_server_hook.py: HTTPServer(("", 8080)), handle_request() loop with timeout = 1). From that point on zero requests reach the new server; every mongod curl call times out until the test fails. On amd64 the first test is the one that fails.

      What is known / not known

      • Not a code change: identical revision passes on macos-14. Identical Python 3.13.15 framework on both fleets.
      • macOS 15 hosts run a windowed GUI session with auto-login (DEVPROD-42555) and a pf firewall (roles/macos/tasks/firewall.yml), neither of which the macos-14 non-GUI hosts had. Suspects: loopback bind/rebind on port 8080 (SO_REUSEADDR / TIME_WAIT after the previous server closes), IPv6-vs-IPv4 resolution of "localhost" in curl vs a wildcard IPv4 bind, or the macOS 15 application firewall prompting for the python binary. None verified yet.
      • Needs a repro on a macos-15 host (spawn host or ask DevProd Build for access to the DEVPROD-44722 fleet). A quick discriminator: bind the hook server to 127.0.0.1 explicitly and/or set allow_reuse_address, and check sudo pfctl -sr on the host.

      Until fixed, the macOS 15 move will carry this as a known failure on enterprise-macos and enterprise-macos-arm64 (2 tasks).

            Assignee:
            Gregory Wlodarek
            Reporter:
            Thomas Elliott-Ward
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: