Windows TLS 1.3 (Schannel): post-handshake NewSessionTicket/KeyUpdate corrupts traffic keys

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fixed
    • Priority: Major - P3
    • 9.0.0-rc0
    • Affects Version/s: None
    • Component/s: None
    • None
    • Server Security
    • Fully Compatible
    • ALL
    • Hide

      On Windows Server 2022 / Windows 11 with TLS 1.3, run the ESE KMIP tests against a PyKMIP server (OpenSSL) with default session tickets enabled. mongod logs SEC_E_DECRYPT_FAILURE and the connection fails with a socket exception

      [RECV_ERROR] ... The specified data could not be decrypted. 

       

      Show
      On Windows Server 2022 / Windows 11 with TLS 1.3, run the ESE KMIP tests against a PyKMIP server (OpenSSL) with default session tickets enabled. mongod logs SEC_E_DECRYPT_FAILURE and the connection fails with a socket exception [RECV_ERROR] ... The specified data could not be decrypted.  
    • Server Security 2026-06-05, Server Security 2026-06-19
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Problem

      On Windows (Schannel) with TLS 1.3, processing a post-handshake message (NewSessionTicket or KeyUpdate) corrupts the application-layer traffic keys, causing the next DecryptMessage to fail with SEC_E_DECRYPT_FAILURE (0x80090330, "The specified data could not be decrypted").

      This breaks Encrypted Storage Engine (ESE) connections to an OpenSSL-backed KMIP server (PyKMIP), which sends two NewSessionTickets immediately after the TLS 1.3 handshake. 

      Root Cause

      When DecryptMessage consumes a post-handshake record it returns SEC_I_RENEGOTIATE (0x00090316) — or, against an OpenSSL peer, the error-severity 0x80090317. Schannel then requires the caller to feed the post-handshake record back into InitializeSecurityContext (client) / AcceptSecurityContext (server) as a SECBUFFER_TOKEN.

      The token must be the undecrypted record in SECBUFFER_EXTRA (buffer index 3). For a consumed NewSessionTicket the decrypted-data buffer SECBUFFER_DATA (index 1) is empty. Passing the empty buffer (or a NULL/empty descriptor) makes ISC/ASC return SEC_E_OK but silently rotate the
      traffic keys with no wire-level KeyUpdate; the peer keeps encrypting with the old key, so the next DecryptMessage fails with SEC_E_DECRYPT_FAILURE.

      The failure is silent (ISC/ASC reports success; the error appears one call later), which previously led to the incorrect conclusion that this was a Schannel bug, worked around by disabling NSTs on the KMIP server (--no-session-tickets / context.num_tickets = 0).

      Documentation

      Microsoft's guidance is split across two pages, which is the source of the ambiguity:

      • DecryptMessage (Schannel): "passing the same buffer as modified by DecryptMessage, ensuring that the SecBuffer type is set to SECBUFFER_TOKEN" — and warns SECBUFFER_EXTRA is not always returned. Ambiguous as to which buffer.
      • Renegotiating an Schannel Connection: "pass the contents of SECBUFFER_EXTRA returned from DecryptMessage in the SECBUFFER_TOKEN." — the authoritative, unambiguous instruction.

      Fix

      • Route both SEC_I_RENEGOTIATE and 0x80090317 through a new helper
        SSLReadManager::processPostHandshakeToken() that feeds the SECBUFFER_EXTRA bytes to ISC/ASC
        as the token, captures ISC/ASC's own trailing SECBUFFER_EXTRA (a following NST or application data) for the next decrypt-loop iteration, and forwards any output token (e.g. a KeyUpdate acknowledgement).
      • Remove the --no-session-tickets workaround from the KMIP test harness.

       

            Assignee:
            Chye Lin Chee
            Reporter:
            Chye Lin Chee
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: