-
Type:
Bug
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: None
-
None
-
Server Security
-
Fully Compatible
-
ALL
-
-
Server Security 2026-06-05, Server Security 2026-06-19
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Problem
On Windows (Schannel) with TLS 1.3, processing a post-handshake message (NewSessionTicket or KeyUpdate) corrupts the application-layer traffic keys, causing the next DecryptMessage to fail with SEC_E_DECRYPT_FAILURE (0x80090330, "The specified data could not be decrypted").
This breaks Encrypted Storage Engine (ESE) connections to an OpenSSL-backed KMIP server (PyKMIP), which sends two NewSessionTickets immediately after the TLS 1.3 handshake.
Root Cause
When DecryptMessage consumes a post-handshake record it returns SEC_I_RENEGOTIATE (0x00090316) — or, against an OpenSSL peer, the error-severity 0x80090317. Schannel then requires the caller to feed the post-handshake record back into InitializeSecurityContext (client) / AcceptSecurityContext (server) as a SECBUFFER_TOKEN.
The token must be the undecrypted record in SECBUFFER_EXTRA (buffer index 3). For a consumed NewSessionTicket the decrypted-data buffer SECBUFFER_DATA (index 1) is empty. Passing the empty buffer (or a NULL/empty descriptor) makes ISC/ASC return SEC_E_OK but silently rotate the
traffic keys with no wire-level KeyUpdate; the peer keeps encrypting with the old key, so the next DecryptMessage fails with SEC_E_DECRYPT_FAILURE.
The failure is silent (ISC/ASC reports success; the error appears one call later), which previously led to the incorrect conclusion that this was a Schannel bug, worked around by disabling NSTs on the KMIP server (--no-session-tickets / context.num_tickets = 0).
Documentation
Microsoft's guidance is split across two pages, which is the source of the ambiguity:
- DecryptMessage (Schannel): "passing the same buffer as modified by DecryptMessage, ensuring that the SecBuffer type is set to SECBUFFER_TOKEN" — and warns SECBUFFER_EXTRA is not always returned. Ambiguous as to which buffer.
- Renegotiating an Schannel Connection: "pass the contents of SECBUFFER_EXTRA returned from DecryptMessage in the SECBUFFER_TOKEN." — the authoritative, unambiguous instruction.
Fix
- Route both SEC_I_RENEGOTIATE and 0x80090317 through a new helper
SSLReadManager::processPostHandshakeToken() that feeds the SECBUFFER_EXTRA bytes to ISC/ASC
as the token, captures ISC/ASC's own trailing SECBUFFER_EXTRA (a following NST or application data) for the next decrypt-loop iteration, and forwards any output token (e.g. a KeyUpdate acknowledgement). - Remove the --no-session-tickets workaround from the KMIP test harness.
- is related to
-
SERVER-79980 Support TLS 1.3 for Windows
-
- Closed
-