[SERVER-60679] Open a port on Mongos that accepts proxied connections and mark Sessions as such Created: 13/Oct/21  Updated: 29/Oct/23  Resolved: 19/Nov/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 5.2.0-rc0, 5.0.7

Type: Improvement Priority: Major - P3
Reporter: Tyler Seip (Inactive) Assignee: Tyler Seip (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
is depended on by SERVER-58496 Detect connections made via the Proxy... Closed
is depended on by DRIVERS-1983 Update load balancer tests to support... Implementing
Related
related to SERVER-74670 A new startup parameter to ignore uns... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v5.0
Sprint: Service Arch 2021-11-15, Service Arch 2021-11-22
Participants:
Story Points: 2

 Description   

The proxy protocol (https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt) requires that all connections made over a given port include the proxy protocol header. In order to support this, open a configurable second port on Mongos that will accept proxied connections and expect the TCP stream to begin with a proxy protocol header, which we should parse. Mark proxied sessions as such.



 Comments   
Comment by Githook User [ 21/Mar/22 ]

Author:

{'name': 'Tyler Seip', 'email': 'Tyler.Seip@mongodb.com', 'username': 'tseip-mongo'}

Message: SERVER-60679: Parse proxy protocol header on proxied connections

(cherry picked from commit 3a18d295d22b377cc7bc4c97bd3b6884d065bb85)
Branch: v5.0
https://github.com/mongodb/mongo/commit/71ba91ab5cdbcd3c7536a9a97c267e1492875b15

Comment by Githook User [ 17/Mar/22 ]

Author:

{'name': 'Tyler Seip', 'email': 'Tyler.Seip@mongodb.com', 'username': 'tseip-mongo'}

Message: SERVER-60679: Parse proxy protocol header on proxied connections

(cherry picked from commit 3a18d295d22b377cc7bc4c97bd3b6884d065bb85)
Branch: LB
https://github.com/mongodb/mongo/commit/a1adce75074f052622ec2fa547999bb65516677c

Comment by Tyler Seip (Inactive) [ 19/Nov/21 ]

For testing these changes, I started a sharded cluster with the load balancing feature flag enabled and the load balancer port set to 10001:

~/mongo/build/opt/install/bin$ ./mongo --nodb
MongoDB shell version v5.2.0-alpha-646-g71e38a8
================
Warning: the "mongo" shell has been superseded by "mongosh",
which delivers improved usability and compatibility.The "mongo" shell has been deprecated and will be removed in
an upcoming release.
For installation instructions, see
https://docs.mongodb.com/mongodb-shell/install/
================
> st = ShardingTest({shards: 1, mongos: 1, mongosOptions: {setParameter: {"featureFlagLoadBalancer": true, "loadBalancerPort": 10001}}});

I then installed and configured haproxy with the following configuration:

 

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
        stats timeout 30s
        user haproxy
        group haproxy
        daemon
 
        # Default SSL material locations                                                                                                                                                                                                                      
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private
        # Default ciphers to use on SSL-enabled listening sockets.                                                                                                                                                                                            
        # For more information, see ciphers(1SSL). This list is from:                                                                                                                                                                                         
        #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/                                                                                                                                                                                  
        # An alternative list with additional directives can be obtained from                                                                                                                                                                                 
        #  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy                                                                                                                                                                     
        ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
        ssl-default-bind-options no-sslv3
defaults
        log     global
        mode    tcp
        option  tcplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http
 
frontend test
bind 127.0.0.1:10000
default_backend mongo-backend
 
backend mongo-backend
balance roundrobin
mode tcp
server mongos1 127.0.0.1:10001 send-proxy-v2

I then connected to the regular mongo port using the mongo shell, just to make sure regular connections still worked:

 

~/mongo/build/opt/install/bin$ ./mongo --port 20004
MongoDB shell version v5.2.0-alpha-646-g71e38a8
connecting to: mongodb://127.0.0.1:20004/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("f5a492ce-5175-4fb6-a036-f64a2a368505") }
MongoDB server version: 5.2.0-alpha-646-g71e38a8
================
Warning: the "mongo" shell has been superseded by "mongosh",
which delivers improved usability and compatibility.The "mongo" shell has been deprecated and will be removed in
an upcoming release.
For installation instructions, see
https://docs.mongodb.com/mongodb-shell/install/
================
---
The server generated these startup warnings when booting:
        2021-11-19T16:06:03.427+00:00: Access control is not enabled for the database. Read and write access to data and configuration is unrestricted
---
mongos> show dbs
admin   0.000GB
config  0.001GB

 

I connected directly to the load balancing port (i.e., not through haproxy) using the shell. We expect this to fail since we aren't emitting a proxy protocol header, and we expect an error to be thrown by the server:

 

~/mongo/build/opt/install/bin$ ./mongo --port 10001
MongoDB shell version v5.2.0-alpha-646-g71e38a8
connecting to: mongodb://127.0.0.1:10001/?compressors=disabled&gssapiServiceName=mongodb
Error: Connection handshake failed. Is your mongod/mongos 3.4 or older? :: caused by :: network error while attempting to run command 'isMaster' on host '127.0.0.1:10001'  :
connect@src/mongo/shell/mongo.js:384:17
@(connect):2:6
exception: connect failed
exiting with code 1

 

along with the corresponding server log message:

s20004| {"t":{"$date":"2021-11-19T16:12:10.878+00:00"},"s":"E",  "c":"NETWORK",  "id":6067900, "ctx":"listener","msg":"Error while parsing proxy protocol header","attr":{"error":"FailedToParse: Initial Proxy Protocol header bytes invalid: \u001d\u0001\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\ufffd\u0007\u0000\u0000\u0001\u0000\u0000\u0000\u0000\u0004\u0001\u0000\u0000\u0010isMaster\u0000\u0001\u0000\u0000\u0000\u0003client\u0000\ufffd\u0000\u0000\u0000\u0003application\u0000\u001d\u0000\u0000\u0000\u0002name\u0000\u000e\u0000\u0000\u0000MongoDB Shell\u0000\u0000\u0003driver\u0000M\u0000\u0000\u0000\u0002name\u0000\u0018\u0000\u0000\u0000MongoDB Internal Client\u0000\u0002version\u0000\u0019\u0000\u0000\u00005.2.0-alpha-646-g71e38a8\u0000\u0000\u0003os\u0000R\u0000\u0000\u0000\u0002type\u0000\u0006\u0000\u0000\u0000Linux\u0000\u0002name\u0000\u0007\u0000\u0000\u0000Ubuntu\u0000\u0002architecture\u0000\u0007\u0000\u0000\u0000x86_64\u0000\u0002version\u0000\u0006\u0000\u0000\u000018.04\u0000\u0000\u0000\u0002$db\u0000\u0006\u0000\u0000\u0000admin\u0000\u0000&7\u001b\r; Make sure your proxy is configured to emit a Proxy Protocol header"}}

Then I connected through the proxy. We expect the connection to succeed, but since I'm using mongo shell, we expect it to be rejected because we don't have load balancing enabled on our driver:

 

~/mongo/build/opt/install/bin$ ./mongo --port 10000
MongoDB shell version v5.2.0-alpha-646-g71e38a8
connecting to: mongodb://127.0.0.1:10000/?compressors=disabled&gssapiServiceName=mongodb
Error: The server is being accessed through a load balancer, but this driver does not have load balancing enabled :
connect@src/mongo/shell/mongo.js:384:17
@(connect):2:6
exception: connect failed
exiting with code 1

Finally, I connect directly to the load balancing port using netcat and supply a fake proxy protocol header just to ensure that the server will hold the connection open properly:

 

~/mongo/build/opt/install/bin$ netcat 127.0.0.1 10001 -N -v -C
Connection to 127.0.0.1 10001 port [tcp/*] succeeded!
PROXY UNKNOWN

 

With the corresponding server log response:

s20004| {"t":{"$date":"2021-11-19T16:15:17.938+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:50872","uuid":"93708e95-d54c-4055-b9be-41a7058db885","connectionId":26,"connectionCount":2}}

And netcatting across the proxy works identically without having to manually supply a proxy header:

 

~/mongo/build/opt/install/bin$ netcat 127.0.0.1 10000 -N -v -C
Connection to 127.0.0.1 10000 port [tcp/webmin] succeeded!

 

s20004| {"t":{"$date":"2021-11-19T16:18:12.598+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:50876","uuid":"b6f70a8c-d2a7-4775-9f3d-308a2f7ea854","connectionId":27,"connectionCount":2}}

Testing finally with v1 proxy protocols being emitted by haproxy (which amounts to changing send-proxy-v2 to send-proxy in the backend configuration of haproxy):

 

~/mongo/build/opt/install/bin$ ./mongo --port 10000
MongoDB shell version v5.2.0-alpha-646-g71e38a8
connecting to: mongodb://127.0.0.1:10000/?compressors=disabled&gssapiServiceName=mongodb
Error: The server is being accessed through a load balancer, but this driver does not have load balancing enabled :
connect@src/mongo/shell/mongo.js:384:17
@(connect):2:6
exception: connect failed
exiting with code 1

which looks good to me.

Comment by Githook User [ 19/Nov/21 ]

Author:

{'name': 'Tyler Seip', 'email': 'Tyler.Seip@mongodb.com', 'username': 'tseip-mongo'}

Message: SERVER-60679: Parse proxy protocol header on proxied connections
Branch: master
https://github.com/mongodb/mongo/commit/3a18d295d22b377cc7bc4c97bd3b6884d065bb85

Generated at Thu Feb 08 05:50:27 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.