[SERVER-65717] Lots of warnings when using X509 replset member authentication Created: 18/Apr/22  Updated: 27/Oct/23  Resolved: 24/Jun/22

Status: Closed
Project: Core Server
Component/s: Logging, Replication, Security
Affects Version/s: 5.0.7
Fix Version/s: None

Type: Bug Priority: Minor - P4
Reporter: Heikki P Assignee: Sergey Galtsev (Inactive)
Resolution: Works as Designed Votes: 0
Labels: SSL, logging, replication, ssl, ssl-certificate, x509
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

5 servers, all Ubuntu 20.04 minimal, 2vcpu, 2Gb ram, 40Gb SSD disks, nothing running except MongoDB


Attachments: PNG File image-2022-04-18-10-35-25-257.png    
Operating System: ALL
Steps To Reproduce:

Install MongoDB 5.0.7

Create self-signed certificates

Init replica set

Restart servers

Sprint: Security 2022-06-13, Security 2022-06-27
Participants:

 Description   

In short

{{}}

 

I’ve created self-signed certificates for our 5-member MongoDB 5.0.7 replica set. I am only running the database, no actual clients (apart from other replica set members of course).

The replica set members can connect and everything seems to work, but we get some certificate related warnings to our logs during startup:

{{}}

Client isn't a mongod or mongos, but is connecting with a certificate with cluster membership

{{}}

{{}}

Client connecting with server's own TLS certificate

{{}}

and after startup,  a..

 

Different user name was supplied to saslSupportedMechs

 

...log message on every authentication between replica set members.

About certificates

Each replset certificate has a certificate with distinct CN=db{nodenum} (eg. CN=db1) set in the certificate's subject field, along with shared values for organisation, organisation unit, address etc.
 
Each node also has /etc/hosts set with hostnames db1,db2,db3,db4,db5.

  • I’ve also tried setting Subject Alt Names in certificates to reflect the same hostnames db1,db2,db3,db4,db5
  • I’ve also tried installing separate certificates to be used as certificateKeyFile and clusterFile }}(with {{extendedKeyUsage set appropriately to serverAuth or clientAuth).

Our Mongo config for TLS and security is:{}{}

tls:
  mode: preferTLS
  certificateKeyFile: /etc/ssl/db1.pem # 2,3,4,5 for others
  clusterFile: /etc/ssl/cluster_db1.pem # 2,3,4,5 for others
  CAFile: /etc/ssl/ca.pem
  allowConnectionsWithoutCertificates: true
security:
  authorization: enabled
  clusterAuthMode: x509

{{}}

{{}}

I can get the replica set to work with all of these variations, but I cannot get these certificate warnings to go away:

{{}}


{{

{"t":\{"$date":"2022-04-17T08:39:51.251+02:00"}

,"s":"W", "c":"ACCESS", "id":20430, "ctx":"conn19","msg":"Client isn't a mongod or mongos, but is connecting with a certificate with cluster membership"}}}


{{{"t":

{"$date":"2022-04-17T08:39:51.216+02:00"}

,"s":"W", "c":"NETWORK", "id":23236, "ctx":"conn11","msg":"Client connecting with server's own TLS certificate"}}}

 

When inspecting the log stream more closely:

 

{"t":{"$date":"2022-04-17T22:31:42.832+02:00"},"s":"I", "c":"REPL", "id":40440, "ctx":"initandlisten","msg":"Starting the TopologyVersionObserver"}
 
{"t":{"$date":"2022-04-17T22:31:42.832+02:00"},"s":"I", "c":"REPL", "id":40445, "ctx":"TopologyVersionObserver","msg":"Started TopologyVersionObserver"}
 
{"t":{"$date":"2022-04-17T22:31:42.833+02:00"},"s":"I", "c":"NETWORK", "id":23015, "ctx":"listener","msg":"Listening on","attr":{"address":"/tmp/mongodb-27017.sock"}} {"t":{"$date":"2022-04-17T22:31:42.833+02:00"},"s":"I", "c":"NETWORK", "id":23015, "ctx":"listener","msg":"Listening on","attr":{"address":"10.0.1.1"}}
 
{"t":{"$date":"2022-04-17T22:31:42.833+02:00"},"s":"I", "c":"NETWORK", "id":23015, "ctx":"listener","msg":"Listening on","attr":{"address":"127.0.0.1"}}
 
{"t":{"$date":"2022-04-17T22:31:42.833+02:00"},"s":"I", "c":"NETWORK", "id":23016, "ctx":"listener","msg":"Waiting for connections","attr":{"port":27017,"ssl":"on"}}
 
{"t":{"$date":"2022-04-17T22:31:42.833+02:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.1.1:57554","connectionId":2,"connectionCount":1}}
 
{"t":{"$date":"2022-04-17T22:31:42.851+02:00"},"s":"W", "c":"NETWORK", "id":23236, "ctx":"conn2","msg":"Client connecting with server's own TLS certificate"}
 
{"t":{"$date":"2022-04-17T22:31:42.852+02:00"},"s":"I", "c":"ACCESS", "id":5286202, "ctx":"conn2","msg":"Different user name was supplied to saslSupportedMechs","attr":{"error":{"code":17,"codeName" :"ProtocolError","errmsg":"Attempt to switch database target during SASL authentication."}}}
 
{"t":{"$date":"2022-04-17T22:31:42.852+02:00"},"s":"W", "c":"ACCESS", "id":20430, "ctx":"conn2","msg":"Client isn't a mongod or mongos, but is connecting with a certificate with cluster membership "}
 
{"t":{"$date":"2022-04-17T22:31:42.852+02:00"},"s":"I", "c":"ACCESS", "id":20429, "ctx":"conn2","msg":"Successfully authenticated","attr":{"client":"10.0.1.1:57554","mechanism":"MONGODB-X509","use r":"CN=db1,OU=MONGO_CLUSTER,O=MY_ORG,L=MY_CITY,ST=MY_STATE,C=MY_COUNTRY","db":"$external"}}
 
{"t":{"$date":"2022-04-17T22:31:42.853+02:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn2","msg":"Connection ended","attr":{"remote":"10.0.1.1:57554","connectionId":2,"connectionCount":0}}

 

...and to me it seems that the db1 (in this case) is connecting with itself, and then complaining about a client (itself?) using the server's own certificate for authenticating, and then complaining about the client not being a mongod or mongos (which it definitely is). All very confusing for this less educated MongoDB user.

Possibly relevant

After the initial startup, we also get a constant stream of these log messages (always in this order)

 

{"t":{"$date":"2022-04-18T09:33:20.115+02:00"},"s":"I",  "c":"ACCESS",   "id":5286202, "ctx":"conn45","msg":"Different user name was supplied to saslSupportedMechs","attr":{"error":{"code":17,"codeName":"ProtocolError","errmsg":"Attempt to switch database target during SASL authentication."}}}

{"t":{"$date":"2022-04-18T09:33:20.116+02:00"},"s":"I",  "c":"ACCESS",   "id":20429,   "ctx":"conn45","msg":"Successfully authenticated","attr":{"client":"10.0.1.5:53270","mechanism":"MONGODB-X509","user":"CN=db5,OU=MONGO_CLUSTER,O=MY_ORG,L=MY_CITY,ST=MY_STATE,C=US","db":"$external"}}

 

...I have no idea whether they are related to the certificate warning or not, and if they should be dealt with somehow. It would perhaps help understanding what is going on, if the log message also shared details about the relevant databases and user names?

Anyway, any help in shedding some light to this would be much appreciated, bug or not.

{}{}



 Comments   
Comment by Oleg Rekutin [ 13/Sep/23 ]

ilari@tahtoo.fi "Attempt to switch database target during SASL Authentication" has been fixed in SERVER-70242

Comment by Sergey Galtsev (Inactive) [ 23/Jun/22 ]

ilari@tahtoo.fi please indicate whether you are interested in further investigation, otherwise I shall be closing the ticket shortly

Comment by Sergey Galtsev (Inactive) [ 09/Jun/22 ]

Hi ilari@tahtoo.fi. Can you please describe in details how are you launching the cluster? Some apps used in testing, namely mtools actually attempt to connect to cluster, which gets logged. You will not see it in 5.x, but in 6.0 you can see client metadata in log file (id: 51800), and if it isn't mongod or mongos, chances are that something else is connecting. Log entry might look as such (irrelevant information removed):

{"t":{"$date":"..."},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn2","msg":"client metadata","attr":{"remote":"127.0.0.1:38970","client":"conn2","doc":{"application":{"name":"mongod","pid":"..."},"driver":{"name":"NetworkInterfaceTL-ReplNetwork","version":"6.1.0-alpha-1060-g131ea34"},"os":{ ... }}}}

Please retest using a 6.0 executable and let us know what the logs show. If you don't mind, please do attach full logs to the ticket, not just screenshots, as we need to be able to analyze them.

Comment by Chris Kelly [ 24/May/22 ]

Hello Heikki,

Thanks for your patience on this. After further investigation, this issue looks like it exists in at least 4.2.19, 4.4.8, and 5.0.7 after testing. It sounds like this is not intended, so I'll forward this to the Security team to take a look at this logging behavior. I was also able to replicate this without any clients involved.

Thanks for your report, and the additional information on this!

 

Comment by Heikki P [ 12/May/22 ]

Thank you for the reply.

We are not using MONGODB-X509 authentication for users, only the SCRAM (username / password) -authentication mechanism, except for the replica set members, which (to my understanding) are using MONGODB-X509 for authentication when `clusterAuthMode: x509` is set, like we have.

Also, we actually did use different values for OU in our client and server certificates (but same O), and we didn't specify DC at all.

I attempted to reduce the problem to it's most minimal form:

  • I made sure client certificate and the server certificates differ in all of the O/OU/CN -parts
  • I removed the configured `clusterFile` certificate altogether for simplicity's sake
  • I don't even start any of the clients, just the replica set

Here are the certificates' subject fields:

 

# client.pem
subject=/C=US/ST=NY/L=NEW YORK/O=MyOrg Clients/OU=MONGO_CLIENT/CN=client
 
# db1.pem, db2.pem, db3.pem, db4.pem, db4.pem
subject=/C=US/ST=NY/L=NEW YORK/O=MyOrg DB Server/OU=MONGO_SERVER/CN=db1
subject=/C=US/ST=NY/L=NEW YORK/O=MyOrg DB Server/OU=MONGO_SERVER/CN=db2
subject=/C=US/ST=NY/L=NEW YORK/O=MyOrg DB Server/OU=MONGO_SERVER/CN=db3
subject=/C=US/ST=NY/L=NEW YORK/O=MyOrg DB Server/OU=MONGO_SERVER/CN=db4
subject=/C=US/ST=NY/L=NEW YORK/O=MyOrg DB Server/OU=MONGO_SERVER/CN=db5
 

 

Unless I'm understanding things backwards, this should fulfil the requirements?] ?

  • A single Certificate Authority (CA) must issue all the x.509 certificates for the members of a sharded cluster or a replica set.
  • The Distinguished Name (DN), found in the member certificate's subject, must specify a non-empty value for at least one of the following attributes:
    • the Organization (O)
    • the Organizational Unit (OU)
    • the Domain Component (DC)
  • The Organization attributes (O's), the Organizational Unit attributes (OU's), and the Domain Components (DC's) must match those from both the net.tls.clusterFile andnet.tls.certificateKeyFile certificates for the other cluster members (or the tlsX509ClusterAuthDNOverride value, if set).

To match, the certificate must match all specifications of these attributes, even the non-specification of these attributes. The order of the attributes does not matter.

In the following example, the two DN's contain matching specifications for O, OU as well as the non-specification of the DC attribute.

 

Also, this requirement specified is fulfilled:

Only cluster member x509 certificates should use the same O, OU, and DC attribute combinations.

Even with this setup, when the replica set starts, logs get these warnings:

 

{"t":{"$date":"2022-05-11T18:16:10.638+02:00"},"s":"W",  "c":"ACCESS",   "id":20430,   "ctx":"conn16","msg":"Client isn't a mongod or mongos, but is connecting with a certificate with cluster membership"}
{"t":{"$date":"2022-05-11T18:16:10.640+02:00"},"s":"W",  "c":"ACCESS",   "id":20430,   "ctx":"conn17","msg":"Client isn't a mongod or mongos, but is connecting with a certificate with cluster membership"}
{"t":{"$date":"2022-05-11T18:16:10.655+02:00"},"s":"W",  "c":"ACCESS",   "id":20430,   "ctx":"conn18","msg":"Client isn't a mongod or mongos, but is connecting with a certificate with cluster membership"}
{"t":{"$date":"2022-05-11T18:16:10.886+02:00"},"s":"W",  "c":"ACCESS",   "id":20430,   "ctx":"conn19","msg":"Client isn't a mongod or mongos, but is connecting with a certificate with cluster membership"}

 

 

Comment by Chris Kelly [ 11/May/22 ]

Hi Heikki,

After looking into your issue, it appears this is likely working as intended. Specifically, the documentation provides guidance on using x509 certificates with MongoDB.

If a client x.509 certificate's subject matches the O, OU, and DC attributes of the Member x.509 Certificate (or tlsX509ClusterAuthDNOverride, if set) exactly, the client connection is accepted, full permissions are granted, and a warning message appears in the log.

Only cluster member x509 certificates should use the same O, OU, and DC attribute combinations.

The following warnings are attributable to this issue:

Client connecting with server's own TLS certificate
Client isn't a mongod or mongos, but is connecting with a certificate with cluster membership

This warning message can show up when all of the following are true:

  • the cluster is using clusterAuthMode: x509
  • the authentication mechanism MONGODB-X509 is enabled on the cluster
  • a user has a certificateKeyFile that is signed by a CA that is valid according to the server's CAFile (clusterCAFile if that is defined)
  • the certificate in the certificateKeyFile has matching O/OU-DC with the server's certificate

Because this sounds like what you have set up currently, it is likely this is the issue and you may want to consider changing the O/OU-DC between the certificateKeyFile and the server's certificate.

I'm going to close this one for now, but if you have further questions, feel free to check out our MongoDB Developer Community Forums for more help.

If the discussion there leads you to suspect a bug in the MongoDB server, then we'd want to investigate it as a possible bug here in the SERVER project.

Regards,
Christopher

Generated at Thu Feb 08 06:03:26 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.