[GODRIVER-2557] unable to connect to ReplicaSetNoPrimary through envoy-proxy Created: 26/Sep/22  Updated: 27/Oct/23  Resolved: 12/Nov/22

Status: Closed
Project: Go Driver
Component/s: None
Affects Version/s: 1.10.2
Fix Version/s: None

Type: Bug Priority: Unknown
Reporter: Nicolaus Maclavellus Assignee: Matt Dale
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Summary

Unable to connect though an envoy proxy to a mongodb cluster.

Mongodb: ReplicaSetNoPrimary.

Driver: 1.10.2

How to Reproduce

The desiderata configuration on the application we would like to obtain is 

DB_MONGO_URL='mongodb://localhost:27017/database?authMechanism=MONGODB-X509' 

All the certificates/keys should be managed by the envoy-proxy.

 

Envoy version:

docker.io/envoyproxy/envoy:v1.16.0

 

We are getting a timeout connecting.

Additional Background

Yaml configuration for envoy

static_resources:
  listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 27017
    filter_chains:
    - filters:
      - name: envoy.mongo_proxy
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.mongo_proxy.v3.MongoProxy
          stat_prefix: mongo_proxy
      - name: envoy.tcp_proxy
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
          stat_prefix: tcp_proxy
          cluster: mongo
          idle_timeout: 7300s
  clusters:
  - name: mongo
    connect_timeout: 0.25s
    type: strict_dns
    lb_policy: round_robin
    hosts:
    - socket_address:
        address: mongotest
        port_value: 27017
    transport_socket:
      name: envoy.transport_sockets.tls
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
        common_tls_context:
          tls_certificates:
            certificate_chain: { "filename": "/etc/envoy/mongo.pem" }
            private_key: { "filename": "/etc/envoy/mongo.pem" }
          validation_context:
            trusted_ca:
              filename: /etc/envoy/ca.crt
admin:
  access_log_path: "/dev/null"
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 8001
 



 Comments   
Comment by Matt Dale [ 12/Nov/22 ]

nicolaoscw@gmail.com thanks for the extra info and sorry for the slow response! I believe that the issue you're encountering is due to the requirement that the MongoDB driver must be able to address the nodes in a replica set with the same DNS names that the replica set nodes use to address each other.

For example, if you have a 3-node replica set where the node DNS names and ports are:

  • node1.example.com:27017
  • node2.example.com:27017
  • node3.example.com:27017

When the driver connects to one of the nodes in the replica set, that node returns a description of the replica set, including the DNS names for each node, which the driver then attempts to connect to. The driver must be able to connect to node1.example.com:27017, node2.example.com:27017, and node3.example.com:27017 using those DNS names. The same is true when connecting through a proxy. As far as I'm aware, the Envoy proxy with the MongoDB plugin is not currently capable of that kind of routing. (Note that it may be possible to run one Envoy proxy per replica set node at the database side instead of at the client side, but I'm not sure if that works for your use case and doesn't seem to be officially supported by Envoy proxy.)

Connecting to a MongoDB sharded cluster does seem to be supported by Envoy proxy because connections from the driver to sharded clusters do not have the same addressability requirements as connections to replica sets.

TLDR: Connecting to a MongoDB replica set through Envoy proxy is not supported because of the addressability requirements when connecting to MongoDB replica sets. Only connecting to MongoDB sharded clusters (and standalone nodes) through Envoy proxy seems to be supported. Note that these requirements and are not specific to the Go Driver but are required for any driver connecting to a MongoDB replica set.

Since this is a limitation of Envoy proxy and is not caused by a Go Driver bug, I'm closing this ticket. Please feel free to open another ticket if you have additional questions. Thanks!

Comment by Nicolaus Maclavellus [ 13/Oct/22 ]

1- confirm, 3 nodes + arbiters

2 - no

3 - 4.4.1

4 - yes

5 - It seems i am able to connect through envoy, this on my pc. The resolution of hostnames of other cluster nodes is working tho. Do you need an effective test with mongosh from a docker image?

 

// code mongosh "mongodb://localhost:27017/mydb?authMechanism=MONGODB-X509&directConnection=true"
Current Mongosh Log ID: 6347d069d56a76d5fc624558
Connecting to:          mongodb://localhost:27017/mydb?authMechanism=MONGODB-X509&directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+1.6.0
Using MongoDB:          4.4.1
Using Mongosh:          1.6.0For mongosh info see: https://docs.mongodb.com/mongodb-shell/
To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy).
You can opt-out by running the disableTelemetry() command.MongoProdReplSet [direct: primary] mydb> db.runCommand({hello:1})
MongoServerError: no such command: 'hello' 

 

 

Comment by Matt Dale [ 11/Oct/22 ]

nicolaoscw@gmail.com thanks for the additional information! I think the error you're experiencing is related to the way that MongoDB drivers discover hosts in a MongoDB replica set. When a driver performs a handshake with a MongoDB replica set using the host list in the connection string, the MongoDB server includes a list of all known hosts in the replica set. If those returned hostnames cannot be resolved by the driver, then it can't connect to the rest of the nodes in the replica set, which leads to operation errors.

I have some questions to help me troubleshoot the issue:

  1. So far I'm assuming you are trying to connect to a MongoDB replica set. Is that accurate? If not, what is the topology of your MongoDB deployment (i.e. a replica set, sharded cluster, or standalone)?
  2. Is your MongoDB deployment running in Kubernetes?
  3. What version of MongoDB are you running?
  4. Is your application using the Go Driver running in Kubernetes?
  5. Are you able to connect to your MongoDB replica set using either the mongosh command line tool or the legacy mongo command line tool using directConnection=true? If so, please connect and run the following command and paste the output here.

    db.runCommand({hello:1})
    

Comment by Nicolaus Maclavellus [ 04/Oct/22 ]

As at update, it works with the client option: directConnection=true

The performance loss is noticeable though.

Comment by Matt Dale [ 03/Oct/22 ]

Hey nicolaoscw@gmail.com thanks for the ticket, we're looking into it and will respond with more questions or information when we have them!

Generated at Thu Feb 08 08:38:54 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.