[SERVER-10261] Disable SSL session caching on server to avoid Java driver SSL connection problems Created: 19/Jul/13  Updated: 11/Jul/16  Resolved: 06/Nov/13

Status: Closed
Project: Core Server
Component/s: Networking
Affects Version/s: 2.4.5
Fix Version/s: 2.4.7, 2.5.3

Type: Bug Priority: Major - P3
Reporter: Jeffrey Yemin Assignee: Andreas Nilsson
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Package versions:

]$ rpm -qa | grep 'mongo|openssl|sasl' | sort
cyrus-sasl-2.1.23-13.el6_3.1.x86_64
cyrus-sasl-devel-2.1.23-13.el6_3.1.x86_64
cyrus-sasl-gssapi-2.1.23-13.el6_3.1.x86_64
cyrus-sasl-lib-2.1.23-13.el6_3.1.x86_64
cyrus-sasl-plain-2.1.23-13.el6_3.1.x86_64
mongo-10gen-enterprise-2.4.5-mongodb_1.x86_64
mongo-10gen-enterprise-server-2.4.5-mongodb_1.x86_64
openssl-1.0.0-27.el6.x86_64


Issue Links:
Depends
depends on SERVER-9041 proactively detect broken connections... Closed
Related
related to SERVER-9307 Server can abort SSL handshake for cl... Closed
related to SERVER-17022 No SSL Session Caching may not be res... Closed
is related to SERVER-11612 Java driver problems with SSL session... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:

 Description   
Issue Status as of December 12th, 2013

ISSUE SUMMARY
SSL connection caching causing intermittent exceptions for Java clients.

USER IMPACT
Java clients making a high number of consecutive SSL connections to the server sometimes experience connections being shut down due to the session state not having been reset properly.

It is present in versions of MongoDB prior to and including v2.4.6.

SOLUTION
Disable SSL session caching on the server. This also improves the security of the connection since no keys are reused.

WORKAROUNDS
Add a small (ms) timeout between tearing down a connection and opening a new one, to make sure that the old session does not affect the new one.

PATCHES
Production release v2.4.7 contains the fix for this issue, and production release v2.6.0 will contain the fix as well.

Original Description

It's possible to generate SSL handshake errors with a trivial Java program: https://gist.github.com/anonymous/a2c4a8ac8f9e38e22edf. This program loops indefinitely, opening a new SSL socket on each iteration and sending a single write (which initiates the handshake).

It eventually generates this exception: http://cl.ly/image/0A2a0j0L0S1i. Note that the alert descriptions are not consistent, suggesting some sort of corruption.

The number of iterations before an error is not consistent, and it doesn't occur if SSL debugging is enabled in the client.



 Comments   
Comment by Githook User [ 09/Feb/15 ]

Author:

{u'username': u'spencerjackson', u'name': u'Spencer Jackson', u'email': u'spencer.jackson@mongodb.com'}

Message: SERVER-17022 Re-enable SSL caching as it didn't fix SERVER-10261

(cherry picked from commit 74e5e2904304bef4b874c4ba68fe4e6671e1c12b)
Branch: v3.0
https://github.com/mongodb/mongo/commit/656c47bc864f5fa2a1c95fbdd2cf671d7d6ab669

Comment by Githook User [ 09/Feb/15 ]

Author:

{u'username': u'spencerjackson', u'name': u'Spencer Jackson', u'email': u'spencer.jackson@mongodb.com'}

Message: SERVER-17022 Re-enable SSL caching as it didn't fix SERVER-10261

(cherry picked from commit 74e5e2904304bef4b874c4ba68fe4e6671e1c12b)
Branch: v2.6
https://github.com/mongodb/mongo/commit/78d52ff64110f5590e60554c4ba58a68772e61df

Comment by Githook User [ 08/Feb/15 ]

Author:

{u'username': u'spencerjackson', u'name': u'Spencer Jackson', u'email': u'spencer.jackson@mongodb.com'}

Message: SERVER-17022 Re-enable SSL caching as it didn't fix SERVER-10261
Branch: master
https://github.com/mongodb/mongo/commit/74e5e2904304bef4b874c4ba68fe4e6671e1c12b

Comment by Andreas Nilsson [ 06/Nov/13 ]

Consider this ticket closed for 2.4.7 and 2.5.3. Creating a new one for 2.4.9.

Comment by Jeffrey Yemin [ 28/Oct/13 ]

This bug is still reproducible with 2.4.7, though not with 2.5.3.

Comment by auto [ 26/Sep/13 ]

Author:

{u'username': u'milkie', u'name': u'Eric Milkie', u'email': u'milkie@10gen.com'}

Message: SERVER-10261 disable ssl session caching (affects java driver)
Branch: v2.4
https://github.com/mongodb/mongo/commit/9976d1d8e9627010f0741612cbfee1ac6c1562f6

Comment by auto [ 24/Sep/13 ]

Author:

{u'username': u'milkie', u'name': u'Eric Milkie', u'email': u'milkie@10gen.com'}

Message: SERVER-10261 disable ssl session caching (affects java driver)
Branch: master
https://github.com/mongodb/mongo/commit/6fd8a847809b626bf13762045c7b75730c55e784

Comment by Jeffrey Yemin [ 21/Sep/13 ]

It looks like we have to do this server-side, cailin.nelson@10gen.com.

Comment by Andreas Nilsson [ 21/Sep/13 ]

One option could be to disable session caching https://www.openssl.org/docs/ssl/SSL_CTX_set_session_cache_mode.html

Comment by Andreas Nilsson [ 30/Jul/13 ]

1.6.0_51

Comment by Jeffrey Yemin [ 30/Jul/13 ]

What version of Java are you testing with?

Comment by Andreas Nilsson [ 30/Jul/13 ]

ok, noticed that I also get exceptions only less often. It seems to be a race between the client and server closing their sockets. Not sure if the bug is on the client or server side, it seems the Java stack has had some troubles with tearing down connections before. http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6618387

Workarounds:
1. Add a Thread.sleep(1) to sleep one millisec between out.close() and socket.close() in the Java client test seems to avoid the issue.
2. Simply catch the exception on the client side and open a new connection.
3. (untested theory) Turn off Java session caching.

Wireshark dumps indicates that the error occurs if the client closes the socket before it has received a TCP RST from the server. When the next connection is created the Java session caching mechanism will have a messed up state to work with and the error occurs.

Comment by Andreas Nilsson [ 30/Jul/13 ]

Fix does not resolve issue completely.

Comment by auto [ 22/Jul/13 ]

Author:

{u'username': u'milkie', u'name': u'Eric Milkie', u'email': u'milkie@10gen.com'}

Message: SERVER-10261 end TCP sockets cleanly

This is a partial backport of SERVER-9041's commit. In particular, this fixes an issue with
SSL and the Java driver's session caching.
Branch: v2.4
https://github.com/mongodb/mongo/commit/7ea27bae26b61e16439a36431e71d030842acd2c

Comment by Eric Milkie [ 22/Jul/13 ]

Andreas found that SERVER-9041 fixes the issue. Actually, just a small part of it – the part where we added a call to shutdown() on the socket before close() on the fd. We might consider just backporting that piece of it.

The problem appears to be that we are not hanging up the socket as clean as we could be on the server, so on the client side the session cache is left in a non-happy state and the next connection attempt then fails. This would only affect the Java driver because it's the only driver I'm aware of that turns on SSL session caching.

Generated at Thu Feb 08 03:22:42 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.