[SERVER-86025] Windows mongod cannot connect to newer openssl-based KMIP server with --kmipClientCertificateFile, but --kmipClientCertificateSelector works as expected Created: 31/Jan/24  Updated: 05/Feb/24

Status: Open
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Adam Rayner Assignee: Backlog - Security Team
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File mongo_client_to_mongod70.pcapng.pcapng.gz     File mongod_certfile_to_kmip_python310.pcapng     File mongod_certfile_to_kmip_python39_ok.pcapng     File mongod_to_kmip_kmipcertificateselector_ok.pcapng    
Issue Links:
Related
Assigned Teams:
Server Security
Operating System: ALL
Steps To Reproduce:

SETUP

On a Windows Server 2019 Machine, copy over source per the setup guide, install an EA mongod, install the test certs into the cert store:

certutil.exe -addstore -f Root jstests\libs\trusted-ca.pem
certutil.exe -importpfx -f -p qwerty jstests\libs\trusted-client.pfx

 

Edit the kmip_server.py to use @SECLEVEL=2 file at src/monog/db/modules/enterprise/jstest/encrypydb/kmip_server.py:

 

--- a/jstests/encryptdb/kmip_server.py
+++ b/jstests/encryptdb/kmip_server.py
@@ -37,7 +37,7 @@ def patch_server(expected_client_version):
     from kmip.core import policy as operation_policy
     from kmip.core.attributes import CryptographicParameters
-    _OPENSSL_SEC_LEVEL = "@SECLEVEL=1"
+    _OPENSSL_SEC_LEVEL = "@SECLEVEL=2"     
 
def _process_encrypt_patched(self, payload):
         self._logger.info("Processing operation: Encrypt") 

 

 

Start the KMIP server:

C:\python\Python310\python.exe .\src\mongo\db\modules\enterprise\jstests\encryptdb\kmip_server.py 

 

TEST 1

Start mongod using the --kmipClientCertificateFile option:

 ./mongod.exe --tlsMode requireTLS --bind_ip 127.0.0.1 --tlsCertificateKeyFile "z:\mongo\jstests\libs\trusted-server.pem" --tlsCAFile "z:\mongo\jstests\libs\trusted-ca.pem" --enableEncryption --kmipServerName "127.0.0.1" --kmipPort "6666" --kmipServerCAFile "Z:\mongo\jstests\libs\trusted-ca.pem" --encryptionCipherMode "AES256-CBC" --kmipClientCertificateFile "Z:\mongo\jstests\libs\trusted-client.pem" --dbpath "z:\data\db"

Observe this fails to start with the following error:

{"t":{"$date":"2024-01-30T18:30:29.186+00:00"},"s":"E",  "c":"STORAGE",  "id":24248,   "ctx":"initandlisten","msg":"Unable to retrieve key","attr":{"keyId":".system","error":{"code":9001,"codeName":"SocketException","errmsg":"socket exception [RECV_ERROR] server [The client and server cannot communicate, because they do not possess a common algorithm.]"}}} 

TEST 2

Start mongod using the --kmipClientCertificateSelector option:

PS C:\Program Files\MongoDB\Server\7.0\bin>  ./mongod.exe --tlsMode requireTLS --bind_ip 127.0.0.1 --tlsCertificateKeyFile "z:\mongo\jstests\libs\trusted-server.pem" --tlsCAFile "z:\mongo\jstests\libs\trusted-ca.pem" --enableEncryption --kmipServerName "127.0.0.1" --kmipPort "6666" --kmipServerCAFile "Z:\mongo\jstests\libs\trusted-ca.pem" --encryptionCipherMode "AES256-CBC" --kmipClientCertificateSelector "thumbprint=0a3ceff09ffc8f5978b32666f039d8e2c061bb3a" --dbpath "z:\data\db"

 

Observe that the server starts successfully.

NOTES

Disclaimer: I'm neither a Windows person nor versed in C++ networking code enough to be trusted here, but my thoughts so far:

  • I notice that we use different connection and SChannel logic when the server is listening for incoming connections 

    src/mongo/util/net/ssl/impl 
    src/mongo/util/net/ssl/detail
    etc.

     versus when it connects as a client (as we do here) - the difference in SChannel configurations or APIs may be worth investigating further. 

  • Most notably we use an in-memory keystore when using --kmipClientCertificateFile versus the windows keystore with --kmipClientCertificateSelector
  • We also pipe SChannel code into ASIO SChannel code for handling TLS handshakes and such, maybe there is some default settings or other things getting baked into the client connection handling
Participants:

 Description   

A mongod running in Windows will fail to connect to an OpenSSL-based KMIP server using Security Level 2, when running with -kmipClientCertificateFile, but will SUCCEED when using -kmipClientCertificateSelector

 

This ticket will investigate and identify the root cause of this discrepancy and, possibly, provide a fix.


Generated at Thu Feb 08 06:59:13 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.