[SERVER-67008] Cannot user client certificate in cluster when "tlsClusterCAFile" is used Created: 03/Jun/22  Updated: 27/Oct/23  Resolved: 10/Jun/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 5.0.9
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Wernfried Domscheit Assignee: Chris Kelly
Resolution: Works as Designed Votes: 0
Labels: TLS/SSL, cluster, replica-set
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:

This is the relevant part of the config file:

 

net:
  port: 27037
  bindIpAll: true
  tls:
    mode: preferTLS
    allowConnectionsWithoutCertificates: true
    certificateKeyFile: c:\MongoDB\config\mongo.server.pem
    CAFile: c:\MongoDB\config\mongo-ca.cer
    clusterFile: c:\MongoDB\config\mongo.member.pem
    clusterCAFile: c:\MongoDB\config\mongo.member-ca.cer
security:
  clusterAuthMode: x509
  authorization: enabled 

 

 

Relevant information of the certificates:

 

openssl x509 -in c:\MongoDB\config\mongo-ca.cer -noout -subject
subject=C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = OSS, CN = mongo-ca
 
openssl x509 -in c:\MongoDB\config\mongo.member-ca.cer -noout -subject
subject=C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = OSS, CN = mongo.member-ca
 
openssl x509 -in c:\MongoDB\config\mongo.member.pem -noout -issuer -subject -ext extendedKeyUsage,keyUsage
issuer=C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = OSS, CN = mongo.member-ca
subject=C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = mongodb.member, CN = mongodb.member
X509v3 Key Usage: critical
    Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
    TLS Web Client Authentication
 
openssl x509 -in c:\MongoDB\config\mongo.server.pem -noout -issuer -subject -ext extendedKeyUsage,keyUsage
issuer=C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = OSS, CN = mongo-ca
subject=C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = mongodb.server, CN = mongodb.server
X509v3 Key Usage: critical
    Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
    TLS Web Server Authentication 

Certificates are valid:

openssl verify -CAfile mongo.member-ca.cer -verify_name ssl_client mongo.member.pem
mongo.member.pem: OK
 
openssl verify -CAfile mongo-ca.cer -verify_name ssl_server mongo.server.pem
mongo.server.pem: OK
 

 

Start Replica Set members. When the second member starts then I get this error on the first one: 

{
    "t": { "$date": "2022-06-03T15:32:15.267+02:00"    },
    "s": "I",
    "c": "ACCESS",
    "id": 20428,
    "ctx": "conn898",
    "msg": "Failed to authenticate",
    "attr": {
        "client": "10.80.41.19:64138",
        "mechanism": "MONGODB-X509",
        "user": "CN=mongodb.member,OU=mongodb.member,O=Sunrise-UPC,L=Zurich,ST=ZH,C=CH",
        "db": "$external",
        "error": {
            "code": 11,
            "codeName": "UserNotFound",
            "errmsg": "Could not find user \"CN=mongodb.member,OU=mongodb.member,O=Sunrise-UPC,L=Zurich,ST=ZH,C=CH\" for db \"$external\""
        }
    }
} 

and member remains in SECONDARY state. Apparently the replica set considers the member connection as a normal client connection, which is not the case. (I am not try to connect with any client)

 

I tied with different certificates (matching O and OU):

openssl x509 -in c:\MongoDB\config\mongo.member.pem -noout -issuer -subject -ext extendedKeyUsage,keyUsage
issuer=C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = OSS, CN = mongo.member-ca
subject=C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = OSS, CN = mongodb.member
X509v3 Key Usage: critical
    Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
    TLS Web Client Authentication
 
openssl x509 -in c:\MongoDB\config\mongo.server.pem -noout -issuer -subject -ext extendedKeyUsage,keyUsage
issuer=C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = OSS, CN = mongo-ca
subject=C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = OSS, CN = mongodb.server
X509v3 Key Usage: critical
    Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
    TLS Web Server Authentication

With these certificates the ReplicaSet starts up as expected:

{
    "t": { "$date": "2022-06-03T15:57:21.081+02:00"    },
    "s": "I",
    "c": "ACCESS",
    "id": 20429,
    "ctx": "conn3",
    "msg": "Successfully authenticated",
    "attr": {
        "client": "10.80.41.19:51039",
        "mechanism": "MONGODB-X509",
        "user": "CN=mongodb.member,OU=OSS,O=Sunrise-UPC,L=Zurich,ST=ZH,C=CH",
        "db": "$external"
    }
} 

However, I get this warning which actually no problem - but it is not true. The client is another mongod repliaset member, thus this warning should not appear:

{
    "t": {"$date": "2022-06-03T15:57:21.081+02:00" },
    "s": "W",
    "c": "ACCESS",
    "id": 20430,
    "ctx": "conn3",
    "msg": "Client isn't a mongod or mongos, but is connecting with a certificate with cluster membership"
}
 

Anyway, I tried several client certificates, none of them is working:

openssl x509 -in c:\MongoDB\config\mongo.client.pem -noout -issuer -subject -ext extendedKeyUsage,keyUsage
issuer=C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = OSS, CN = mongo-ca
subject=C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = ClientAuthentication, CN = admin
X509v3 Key Usage: critical
    Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
    TLS Web Client Authentication
 
openssl x509 -in mongo.client.pem -noout -subject -nameopt RFC2253
subject=CN=admin,OU=ClientAuthentication,O=Sunrise-UPC,L=Zurich,ST=ZH,C=CH
 
db.getSiblingDB("$external").runCommand({
   createUser: "CN=admin,OU=ClientAuthentication,O=Sunrise-UPC,L=Zurich,ST=ZH,C=CH",
   roles: [{ role: "root", db: "admin" }]
})
 

I cannot use the client certificate, neither for TLS/SSL nor for authentication:

mongosh --norc --quiet "mongodb://localhost:27037/admin?authSource=$external" --tls --tlsCertificateKeyFile=mongo.client.pem --tlsCAFile=mongo-ca.cer --authenticationMechanism=MONGODB-X509
MongoServerSelectionError: connection <monitor> to 127.0.0.1:27037 closed
 
mongosh --norc --quiet "mongodb://admin:password@localhost:27037/admin?authSource=admin" --tls --tlsCertificateKeyFile=mongo.client.pem --tlsCAFile=mongo-ca.cer
MongoServerSelectionError: connection <monitor> to 127.0.0.1:27037 closed
 

Note, the same works fine on a Standalone MongoDB, it only fails on a cluster/replica set:

mongosh --norc --quiet "mongodb://localhost:27017/admin?authSource=$external" --eval "db.getMongo()" --tls --tlsCertificateKeyFile=mongo.client.pem --tlsCAFile=mongo-ca.cer --authenticationMechanism=MONGODB-X509
mongodb://localhost:27017/admin?authSource=%24external&directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+1.3.1
 
mongosh --norc --quiet "mongodb://admin:password@localhost:27017/admin?authSource=admin" --eval "db.getMongo()" --tls --tlsCertificateKeyFile=mongo.client.pem --tlsCAFile=mongo-ca.cer
mongodb://<credentials>@localhost:27017/admin?authSource=admin&directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+1.3.1
 

It also works fine if I use a common CA for cluster and client, i.e. 

net:
  port: 27037
  bindIpAll: true
  tls:
    mode: preferTLS
    allowConnectionsWithoutCertificates: true
    certificateKeyFile: c:\MongoDB\config\mongo.server.pem
    CAFile: c:\MongoDB\config\mongo-ca.cer
    clusterFile: c:\MongoDB\config\mongo.member.pem
 

works fine. However, I like to have different CA for cluster and client certificates.

Is this a bug or do I anything wrong?

 

Participants:

 Description   

I am running some tests with TLS/SSL certificates. I like to use client certificates (for TLS/SSL and authentication) in a ReplicaSet / Sharded Cluster.

It works fine when I run it on standalone MongoDB

It works fine when I use a common CA for cluster and client, i.e.

net:
  port: 27037
  bindIpAll: true
  tls:
    mode: preferTLS
    allowConnectionsWithoutCertificates: true
    certificateKeyFile: c:\MongoDB\config\mongo.server.pem
    CAFile: c:\MongoDB\config\mongo-ca.cer
    clusterFile: c:\MongoDB\config\mongo.member.pem
 

works fine. However, I like to have different CA for cluster and client certificates.



 Comments   
Comment by Chris Kelly [ 10/Jun/22 ]

Hi Wernfried,

Thanks for taking the time to come back and share the extra information on that. I'll go ahead and close this ticket.

Regards,
Christopher

Comment by Wernfried Domscheit [ 08/Jun/22 ]

After creating dozens of different certificates combinations I got it.

clusterCAFile is used to verify certificates provided by the client - "normal" client as well as "membership" client connections.

In my opinion the name is a bit misleading, I was thinking it is related to clusterFile, i. e.

  • clusterCAFile is used to verify certificates from "membership" clients
  • CAFile is used to verify certificates from "normal" clients

Obviously, my assumption was wrong (although I don't see any use-case for the way it is implemented)

You can close this ticket.

 

Kind Regards
Wernfried

 

 

Comment by Wernfried Domscheit [ 08/Jun/22 ]

Made a few more test. Obviously the client-certificate ( --tlsCertificateKeyFile=mongo.client.pem ) is verified by Cluster-CA ( clusterCAFile: mongo.member-ca.cer )

 

set EVAL=db.runCommand({ connectionStatus: 1 }).authInfo.authenticatedUsers
 
c:\MongoDB\config>openssl verify -CAfile mongo-ca.cer -show_chain -verify_name ssl_client mongo.client.pem
mongo.client.pem: OK
Chain:
depth=0: C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = OSS, OU = MongoDB, OU = Client, CN = admin (untrusted)
depth=1: C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = OSS, CN = mongo-ca
 
mongosh --norc --quiet "mongodb://admin:password@%HOST%/admin?authSource=admin&replicaSet=repSet" --eval "%EVAL%" --tls --tlsCAFile=mongo-ca.cer --tlsCertificateKeyFile=mongo.client.pem
MongoServerSelectionError: connection <monitor> to 127.0.0.1:27037 closed
 
 
 
 
c:\MongoDB\config>openssl verify -CAfile mongo.member-ca.cer -show_chain -verify_name ssl_client mongo.client-x.pem
mongo.client-x.pem: OK
Chain:
depth=0: C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = OSS, OU = MongoDB, OU = Client, CN = admin (untrusted)
depth=1: C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = OSS, CN = mongo.member-ca 
 
mongosh --norc --quiet "mongodb://admin:password@%HOST%/admin?authSource=admin&replicaSet=repSet" --eval "%EVAL%" --tls --tlsCAFile=mongo-ca.cer --tlsCertificateKeyFile=mongo.client-x.pem
[ { user: 'admin', db: 'admin' } ] -> Success

 

 

I read carefully the documentation:

"The .pem file that contains the root certificate chain from the Certificate Authority used to validate the certificate presented by a client establishing a connection. {{}} net.tls.clusterCAFile lets you use separate Certificate Authorities to verify the client to server and server to client portions of the TLS handshake."

 

What does it mean? I don't understand. For "server to client portions of the TLS handshake" the CA is defined at client side ( mongosh --tls --tlsCAFile=... ), so what is then the difference between net.tls.clusterCAFile and net.tls.CAFile?

 

I also tried the opposite

net:
  tls:
    mode: preferTLS
    allowConnectionsWithoutCertificates: true
    CAFile: c:\MongoDB\config\mongo.member-ca.cer
    certificateKeyFile: c:\MongoDB\config\mongo.server.pem
    clusterFile: c:\MongoDB\config\mongo.member.pem
    clusterCAFile: c:\MongoDB\config\mongo-ca.cer
 

But this does not work either, the replica set member cannot connect, the node remains in SECONDARY state.

 

 

 

 

Generated at Thu Feb 08 06:07:02 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.