-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Minor - P4
-
None
-
Affects Version/s: None
-
Component/s: Internal Code
-
Labels:None
-
Service Arch
-
ALL
This concerns a failure when running proxy_protocol_connect.js:
- The test is using two constants for the proxy server and the mongos listening ports (defined here). I'd imagine in some executions, the OS might not be able to allocate the specified ports to either the mongos or the proxy protocol servers.
- The Python wrapper for the ProxyProtocolServer doesn't properly handle exceptions and always succeeds (see here and here). For instance, if the listening port is not available and the proxy server returns an error, the wrapper script is unaware of the failure and ignores the error code.
The logs indicate the ProxyProtocolServer failed to start:
[js_test:proxy_protocol_connect] Proxy protocol server is listening on port: 21234
[js_test:proxy_protocol_connect] Proxy protocol server is proxying to port: 21235
[js_test:proxy_protocol_connect] | 2022-03-29T02:08:34.019Z I - 22810 [js] "shell: Started program","attr":{"pid":"24036","argv":["/data/tmp/build/8e5c2f9da7e47243c5e3d5aa36d5bd0a/venv/bin/python3","-u","jstests/sharding/libs/proxy_protocol_server.py","--service","localhost:21234","localhost:21235?pp=v1"]}
[js_test:proxy_protocol_connect] sh24036| Starting proxy protocol server...
[js_test:proxy_protocol_connect] sh24036| Traceback (most recent call last):
[js_test:proxy_protocol_connect] sh24036| File "/data/tmp/build/8e5c2f9da7e47243c5e3d5aa36d5bd0a/src/jstests/sharding/libs/proxy_protocol_server.py", line 11, in <module>
[js_test:proxy_protocol_connect] sh24036| sys.exit(main())
[js_test:proxy_protocol_connect] sh24036| File "/data/tmp/build/8e5c2f9da7e47243c5e3d5aa36d5bd0a/venv/lib/python3.9/site-packages/proxyprotocol/server/main.py", line 48, in main
[js_test:proxy_protocol_connect] sh24036| return asyncio.run(run(args))
[js_test:proxy_protocol_connect] sh24036| File "/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/python3-v3.UiY/lib/python3.9/asyncio/runners.py", line 44, in run
[js_test:proxy_protocol_connect] sh24036| return loop.run_until_complete(main)
[js_test:proxy_protocol_connect] sh24036| File "/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/python3-v3.UiY/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
[js_test:proxy_protocol_connect] sh24036| return future.result()
[js_test:proxy_protocol_connect] sh24036| File "/data/tmp/build/8e5c2f9da7e47243c5e3d5aa36d5bd0a/venv/lib/python3.9/site-packages/proxyprotocol/server/main.py", line 59, in run
[js_test:proxy_protocol_connect] sh24036| servers = [
[js_test:proxy_protocol_connect] sh24036| File "/data/tmp/build/8e5c2f9da7e47243c5e3d5aa36d5bd0a/venv/lib/python3.9/site-packages/proxyprotocol/server/main.py", line 60, in <listcomp>
[js_test:proxy_protocol_connect] sh24036| await loop.create_server(partial(new_server, dest),
[js_test:proxy_protocol_connect] sh24036| File "/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/python3-v3.UiY/lib/python3.9/asyncio/base_events.py", line 1494, in create_server
[js_test:proxy_protocol_connect] sh24036| raise OSError(err.errno, 'error while attempting '
[js_test:proxy_protocol_connect] sh24036| OSError: [Errno 99] error while attempting to bind on address ('::1', 21234, 0, 0): cannot assign requested address
[js_test:proxy_protocol_connect] Proxy Protocol Server sucessfully started.
Later, then mongo shell attempts to connect to the proxy server as part of running the test, but that attempts fails:
[js_test:proxy_protocol_connect] Error: couldn't connect to server localhost:21234, connection attempt failed: SocketException: Error connecting to localhost:21234 (127.0.0.1:21234) :: caused by :: Connection refused : [js_test:proxy_protocol_connect] testProxyProtocolConnect@jstests/sharding/proxy_protocol_connect.js:29:18 [js_test:proxy_protocol_connect] @jstests/sharding/proxy_protocol_connect.js:62:1 [js_test:proxy_protocol_connect] @jstests/sharding/proxy_protocol_connect.js:6:2 [js_test:proxy_protocol_connect] failed to load: jstests/sharding/proxy_protocol_connect.js [js_test:proxy_protocol_connect] exiting with code -3
To address the issue, we should:
- Properly handle exceptions returned from the proxy server, and potentially handle such exceptions by retrying.
- Allow the OS to assign random, available ports to both servers to avoid similar failures.