-
Type:
Bug
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: None
-
None
-
Storage Engines - Server Integration
-
Fully Compatible
-
ALL
-
SESIWhinery 2026-11-17
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Problem
When the macOS build variants move from the macos-14 to the macos-15 distros (DEVPROD-45657 / SERVER-137597), the queryableWT task fails on both enterprise macOS variants. 3 of the 5 tests in src/mongo/db/modules/enterprise/jstests/queryable_wt/ fail: simple_rs_restore.js, simple_rs_pit_restore.js (arm64) and simple_standalone_restore.js (amd64). The task is green on macos-14 in mainline the same day (0f094cb6, Python 3.13.15 on both fleets).
Evidence: https://spruce.corp.mongodb.com/version/6ad903bc0a71f20007041f5a
- enterprise-macos-arm64 / queryableWT (host mac-m4-1b-1, macos-15-arm64-gui)
- enterprise-macos / queryableWT (host mac1-1a-1, macos-15-amd64-gui)
Symptom
The restored mongod (--storageEngine queryable_wt --queryableBackupApiUri localhost:8080) cannot reach the resmoke-side queryable HTTP server:
"ctx":"CurlConnPool-0","msg":"Connecting","attr":{"hostAndPort":"localhost:8080"}
"msg":"Ending connection due to bad connection status","attr":{"hostAndPort":"localhost:8080","error":"OperationFailed: Bad HTTP response from API server: Timeout was reached"}
... (repeats for ~65s) ...
"c":"WT","msg":"WiredTiger error message","attr":{"message":"wiredtiger_open: [WT_VERB_EXTENSION][ERROR]: ListDir: Bad HTTP response from the ApiServer: Bad HTTP response from API server: Timeout was reached"}
mongo program was not running at 20041, process ended with exit code: 1
In the arm64 run the first test (simple_standalone_restore.js) passes and the server access log shows ~1000 successful /os_list / /os_read / /os_wt_recovery_write requests. QueryableServerHook then stops the server thread after the test and starts a fresh one before the next test (buildscripts/resmokelib/testing/hooks/queryable_server_hook.py: HTTPServer(("", 8080)), handle_request() loop with timeout = 1). From that point on zero requests reach the new server; every mongod curl call times out until the test fails. On amd64 the first test is the one that fails.
What is known / not known
- Not a code change: identical revision passes on macos-14. Identical Python 3.13.15 framework on both fleets.
- macOS 15 hosts run a windowed GUI session with auto-login (DEVPROD-42555) and a pf firewall (roles/macos/tasks/firewall.yml), neither of which the macos-14 non-GUI hosts had. Suspects: loopback bind/rebind on port 8080 (SO_REUSEADDR / TIME_WAIT after the previous server closes), IPv6-vs-IPv4 resolution of "localhost" in curl vs a wildcard IPv4 bind, or the macOS 15 application firewall prompting for the python binary. None verified yet.
- Needs a repro on a macos-15 host (spawn host or ask DevProd Build for access to the DEVPROD-44722 fleet). A quick discriminator: bind the hook server to 127.0.0.1 explicitly and/or set allow_reuse_address, and check sudo pfctl -sr on the host.
Until fixed, the macOS 15 move will carry this as a known failure on enterprise-macos and enterprise-macos-arm64 (2 tasks).