Problem Statement / Rationale

Since the COMPASS-8355 fix ("handle SSH tunnel errors after connection is established", shipped in 1.49.9), devtools-proxy-support responds to a failed SOCKS5 forwarding attempt by marking the error retryable, tearing down the shared SSH client, re-establishing it, and retrying the channel.

That recovery is correct when the SSH client is actually dead (the hibernate case COMPASS-8355 fixed: socket closed, keepalive timeout, Instance unusable after fatal error). It is incorrect for SSH_MSG_CHANNEL_OPEN_FAILURE (e.g. ssh2's (SSH) Channel open failure: No route to host). That reply comes from a healthy sshd on the jump host reporting that one destination is unreachable (EHOSTUNREACH on its outbound connect()). Reconnecting the client can never change the outcome.

The failure mode compounds with replica set discovery: the driver runs a monitor per member through the tunnel, so a single member that is unreachable from the jump host (down, firewalled, stale DNS/hosts entry, unrouted subnet) produces a channel-open failure on every heartbeat (~10s). Each one triggers a teardown/rebuild of the SSH client that the healthy members' channels are riding on.

COMPASS-10804 capped consecutive failures, which stops the unbounded loop, but did not change the classification: per-destination failures still count toward and trigger client-level reconnects. On 1.49.14 the churn recurs on every heartbeat cycle indefinitely.

Regression baseline: the pre-1.49.9 behavior (and the legacy @mongodb-js/ssh-tunnel through 1.43) handled the identical error by closing only the one SOCKS5 socket — log id 1001000265 Error establishing SSH channel for Socks5 request, no client teardown, zero collateral. Side-by-side logs from the same environment on 1.41.0 and 1.49.14 are attached below.

Impact

• Collateral damage to healthy connections: the teardown races in-flight work; the retry frequently lands on the closing client and dies with No response from server (retryableError: false), and healthy members' forwarded sockets can be blipped mid-operation.
• Bastion-side session churn: each rebuild opens a new SSH session on the jump host. This is the same reconnect path implicated in HELP-96049 (300+ leaked sessions exhausting bastion RAM after a session lock).

Steps to Reproduce (minimal)

Setup: a 3-member replica set rs0 (memberA, memberB, memberC, all on :27017) and a jump host J running sshd, where J can reach A and B but not C.

1. On J, make C:27017 host-unreachable. Easiest deterministic simulation — REJECT (not DROP) so sshd gets an immediate EHOSTUNREACH instead of a timeout:

sudo iptables -I OUTPUT -d <memberC-ip> -p tcp --dport 27017 \
     -j REJECT --reject-with icmp-host-unreachable

1. Verify the precondition from J: nc -vz <memberC-ip> 27017 fails with "No route to host" while A and B succeed.

1. In Compass ≥ 1.49.9, create a connection with the SSH tunnel option through J (identity file or password — irrelevant) and connection string:

mongodb://<user>:<pass>@memberA:27017,memberB:27017/?replicaSet=rs0

Note C need not be in the seed list — topology discovery adds it from the primary's hello response, so removing it from the connection string is not a workaround.

1. Connect (succeeds: ReplicaSetWithPrimary, writable) and watch the log for 60 seconds.

Expected Results

The monitor for memberC fails each heartbeat with a request-scoped error; the shared SSH client stays up and channels to A/B are unaffected. This is the observed 1.41.0 behavior — one E line per attempt and nothing else:

{"s":"E","c":"COMPASS-SSH-TUNNEL","id":1001000265,"ctx":"tunnel-0",
 "msg":"Error establishing SSH channel for Socks5 request",
 "attr":{"dstAddr":"memberC","dstPort":27017,
         "error":"Error: (SSH) Channel open failure: No route to host ..."}}
{"s":"W","c":"COMPASS-DATA-SERVICE","id":1001000023,
 "msg":"Server heartbeat failed","attr":{"connectionId":"memberC:27017","failure":"Socket closed"}}

Actual Results (1.49.9+, including 1.49.14 with the COMPASS-10804 cap)

Every heartbeat cycle, the per-destination failure is classified retryable and the shared SSH client is recycled; the retry races the dying client and fails non-retryably:

{"s":"E","c":"DEVTOOLS-PROXY","id":1001000261,"ctx":"ssh-compass-0",
 "msg":"Error forwarding outbound SSH connection, potentially retrying",
 "attr":{"host":"memberC","error":"Error: (SSH) Channel open failure: No route to host ...",
         "retryableError":true,"retriesLeft":1}}
{"s":"I","c":"DEVTOOLS-PROXY","id":1001000257,"ctx":"ssh-compass-0",
 "msg":"Establishing new SSH connection", ...}
{"s":"I","c":"DEVTOOLS-PROXY","id":1001000252,"ctx":"ssh-compass-0","msg":"sshClient closed"}
{"s":"E","c":"DEVTOOLS-PROXY","id":1001000261,"ctx":"ssh-compass-0",
 "msg":"Error forwarding outbound SSH connection, potentially retrying",
 "attr":{"host":"memberC","error":"Error: No response from server",
         "retryableError":false,"retriesLeft":0}}
 
 

…repeating on every subsequent heartbeat (observed at ~10s intervals for the lifetime of the connection). Each cycle tears down and rebuilds the client shared with memberA/memberB.

Proposed Fix

In the SOCKS5 forwarding error handler in devtools-proxy-support, classify ssh2 CHANNEL_OPEN_FAILURE (at minimum reason SSH_OPEN_CONNECT_FAILED; arguably all four reason codes, since "administratively prohibited" is equally non-recoverable by reconnecting) as request-scoped: reply/close only the requesting SOCKS5 socket and do not mark the error retryable or touch the shared client — i.e., restore the pre-1.49.9 semantics for this error class. Reserve the COMPASS-8355 teardown-and-retry path for genuine client-level signals (client socket close/error, keepalive timeout, Instance unusable after fatal error). The COMPASS-10804 cap remains as a backstop for the client-level path.

A sensible regression test: SOCKS5 server over an SSH stub whose forwardOut rejects with a channel-open failure for one destination and succeeds for another; assert the client connection object is never recreated and the healthy destination's channel survives a failed request to the unhealthy one.

Additional Notes / Field Evidence

• Live support case (self-hosted 3-member RS behind a corporate bastion, member 3 unroutable from the bastion): identical environment logged on Compass 1.41.0 and 1.49.14. 1.41 degrades gracefully (member marked Unknown, everything else unaffected); 1.49.14 exhibits the churn above. Full sanitized logs available on request.
• HELP-96049 demonstrates the operational cost of the reconnect path on shared bastions (leaked sessions → RAM exhaustion). This ticket removes an entire class of spurious entries into that path.
• Not a workaround: removing the dead member from the seed list (discovery re-adds it). Actual workarounds today: fix reachability from the jump host, rs.remove() the member, or directConnection=true (loses failover).