Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-86025

Windows mongod cannot connect to newer openssl-based KMIP server with --kmipClientCertificateFile, but --kmipClientCertificateSelector works as expected

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Unresolved
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Server Security
    • ALL
    • Hide

      SETUP

      On a Windows Server 2019 Machine, copy over source per the setup guide, install an EA mongod, install the test certs into the cert store:

      certutil.exe -addstore -f Root jstests\libs\trusted-ca.pem
certutil.exe -importpfx -f -p qwerty jstests\libs\trusted-client.pfx

       

      Edit the kmip_server.py to use @SECLEVEL=2 file at src/monog/db/modules/enterprise/jstest/encrypydb/kmip_server.py:

       

      --- a/jstests/encryptdb/kmip_server.py
+++ b/jstests/encryptdb/kmip_server.py
@@ -37,7 +37,7 @@ def patch_server(expected_client_version):
     from kmip.core import policy as operation_policy
     from kmip.core.attributes import CryptographicParameters
-    _OPENSSL_SEC_LEVEL = "@SECLEVEL=1"
+    _OPENSSL_SEC_LEVEL = "@SECLEVEL=2"     

def _process_encrypt_patched(self, payload):
         self._logger.info("Processing operation: Encrypt")

       

       

      Start the KMIP server:

      C:\python\Python310\python.exe .\src\mongo\db\modules\enterprise\jstests\encryptdb\kmip_server.py

       

      TEST 1

      Start mongod using the --kmipClientCertificateFile option:

       ./mongod.exe --tlsMode requireTLS --bind_ip 127.0.0.1 --tlsCertificateKeyFile "z:\mongo\jstests\libs\trusted-server.pem" --tlsCAFile "z:\mongo\jstests\libs\trusted-ca.pem" --enableEncryption --kmipServerName "127.0.0.1" --kmipPort "6666" --kmipServerCAFile "Z:\mongo\jstests\libs\trusted-ca.pem" --encryptionCipherMode "AES256-CBC" --kmipClientCertificateFile "Z:\mongo\jstests\libs\trusted-client.pem" --dbpath "z:\data\db"

      Observe this fails to start with the following error:

      {"t":{"$date":"2024-01-30T18:30:29.186+00:00"},"s":"E",  "c":"STORAGE",  "id":24248,   "ctx":"initandlisten","msg":"Unable to retrieve key","attr":{"keyId":".system","error":{"code":9001,"codeName":"SocketException","errmsg":"socket exception [RECV_ERROR] server [The client and server cannot communicate, because they do not possess a common algorithm.]"}}}

      TEST 2

      Start mongod using the --kmipClientCertificateSelector option:

      PS C:\Program Files\MongoDB\Server\7.0\bin>  ./mongod.exe --tlsMode requireTLS --bind_ip 127.0.0.1 --tlsCertificateKeyFile "z:\mongo\jstests\libs\trusted-server.pem" --tlsCAFile "z:\mongo\jstests\libs\trusted-ca.pem" --enableEncryption --kmipServerName "127.0.0.1" --kmipPort "6666" --kmipServerCAFile "Z:\mongo\jstests\libs\trusted-ca.pem" --encryptionCipherMode "AES256-CBC" --kmipClientCertificateSelector "thumbprint=0a3ceff09ffc8f5978b32666f039d8e2c061bb3a" --dbpath "z:\data\db"

       

      Observe that the server starts successfully.

      NOTES

      Disclaimer: I'm neither a Windows person nor versed in C++ networking code enough to be trusted here, but my thoughts so far:

      • I notice that we use different connection and SChannel logic when the server is listening for incoming connections  
        src/mongo/util/net/ssl/impl 
src/mongo/util/net/ssl/detail
etc.

         versus when it connects as a client (as we do here) - the difference in SChannel configurations or APIs may be worth investigating further. 

      • Most notably we use an in-memory keystore when using --kmipClientCertificateFile versus the windows keystore with --kmipClientCertificateSelector
      • We also pipe SChannel code into ASIO SChannel code for handling TLS handshakes and such, maybe there is some default settings or other things getting baked into the client connection handling
      Show
      SETUP On a Windows Server 2019 Machine, copy over source per the setup guide, install an EA mongod, install the test certs into the cert store: certutil.exe -addstore -f Root jstests\libs\trusted-ca.pem certutil.exe -importpfx -f -p qwerty jstests\libs\trusted-client.pfx   Edit the kmip_server.py to use @SECLEVEL=2 file at src/monog/db/modules/enterprise/jstest/encrypydb/kmip_server.py:   --- a/jstests/encryptdb/kmip_server.py +++ b/jstests/encryptdb/kmip_server.py @@ -37,7 +37,7 @@ def patch_server(expected_client_version):      from kmip.core import policy as operation_policy      from kmip.core.attributes import CryptographicParameters -    _OPENSSL_SEC_LEVEL = "@SECLEVEL=1" +    _OPENSSL_SEC_LEVEL = "@SECLEVEL=2"       def _process_encrypt_patched(self, payload):          self._logger.info( "Processing operation: Encrypt" )     Start the KMIP server: C:\python\Python310\python.exe .\src\mongo\db\modules\enterprise\jstests\encryptdb\kmip_server.py   TEST 1 Start mongod using the --kmipClientCertificateFile option:  ./mongod.exe --tlsMode requireTLS --bind_ip 127.0.0.1 --tlsCertificateKeyFile "z:\mongo\jstests\libs\trusted-server.pem" --tlsCAFile "z:\mongo\jstests\libs\trusted-ca.pem" --enableEncryption --kmipServerName "127.0.0.1" --kmipPort "6666" --kmipServerCAFile "Z:\mongo\jstests\libs\trusted-ca.pem" --encryptionCipherMode "AES256-CBC" --kmipClientCertificateFile "Z:\mongo\jstests\libs\trusted-client.pem" --dbpath "z:\data\db" Observe this fails to start with the following error: { "t" :{ "$date" : "2024-01-30T18:30:29.186+00:00" }, "s" : "E" ,   "c" : "STORAGE" ,   "id" :24248,   "ctx" : "initandlisten" , "msg" : "Unable to retrieve key" , "attr" :{ "keyId" : ".system" , "error" :{ "code" :9001, "codeName" : "SocketException" , "errmsg" : "socket exception [RECV_ERROR] server [The client and server cannot communicate, because they do not possess a common algorithm.]" }}} TEST 2 Start mongod using the --kmipClientCertificateSelector option: PS C:\Program Files\MongoDB\Server\7.0\bin>  ./mongod.exe --tlsMode requireTLS --bind_ip 127.0.0.1 --tlsCertificateKeyFile "z:\mongo\jstests\libs\trusted-server.pem" --tlsCAFile "z:\mongo\jstests\libs\trusted-ca.pem" --enableEncryption --kmipServerName "127.0.0.1" --kmipPort "6666" --kmipServerCAFile "Z:\mongo\jstests\libs\trusted-ca.pem" --encryptionCipherMode "AES256-CBC" --kmipClientCertificateSelector "thumbprint=0a3ceff09ffc8f5978b32666f039d8e2c061bb3a" --dbpath "z:\data\db"   Observe that the server starts successfully. NOTES Disclaimer: I'm neither a Windows person nor versed in C++ networking code enough to be trusted here, but my thoughts so far: I notice that we use different connection and SChannel logic when the server is listening for incoming connections  src/mongo/util/net/ssl/impl src/mongo/util/net/ssl/detail etc.  versus when it connects as a client (as we do here) - the difference in SChannel configurations or APIs may be worth investigating further.  Most notably we use an in-memory keystore when using --kmipClientCertificateFile versus the windows keystore with --kmipClientCertificateSelector We also pipe SChannel code into ASIO SChannel code for handling TLS handshakes and such, maybe there is some default settings or other things getting baked into the client connection handling

      A mongod running in Windows will fail to connect to an OpenSSL-based KMIP server using Security Level 2, when running with -kmipClientCertificateFile, but will SUCCEED when using -kmipClientCertificateSelector

       

      This ticket will investigate and identify the root cause of this discrepancy and, possibly, provide a fix.

        1. mongod_certfile_to_kmip_python310.pcapng
          13 kB
        2. mongod_to_kmip_kmipcertificateselector_ok.pcapng
          16 kB
        3. mongo_client_to_mongod70.pcapng.pcapng.gz
          19 kB
        4. mongod_certfile_to_kmip_python39_ok.pcapng
          16 kB

            Assignee:
            Unassigned Unassigned
            Reporter:
            adam.rayner@mongodb.com Adam Rayner
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: