[SERVER-16073] Allow disabling SSL Ciphers via hidden flag: sslCipherConfig Created: 11/Nov/14 Updated: 06/Apr/17 Resolved: 13/Mar/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Networking, Security |
| Affects Version/s: | 2.6.4, 2.6.5 |
| Fix Version/s: | 2.6.9, 3.0.3, 3.1.0 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Nemanja Dubravac | Assignee: | Andreas Nilsson |
| Resolution: | Done | Votes: | 2 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
CentOS 6.5 |
||
| Attachments: |
|
||||||||
| Issue Links: |
|
||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||
| Operating System: | Linux | ||||||||
| Backport Completed: | |||||||||
| Participants: | |||||||||
| Case: | (copied to CRM) | ||||||||
| Description |
|
Issue Status as of Mar 17, 2015 ISSUE SUMMARY USER IMPACT WORKAROUNDS
AFFECTED VERSIONS FIX VERSION RESOLUTION DETAILS Original descriptionwe're using mongodb (tried both 2.6.4 & 2.6.5) compiled with ssl on our own, and we're having problems recovering servers when they have to do a complete (initial) sync
after it fails multiple times, it finally gives up the initial sync and shuts down we could reproduce the same behavior on another server in a different shard of the same cluster we're using openssl 1.0.1e 30.el6_6.4 on our mongod instances and the same version was used while compiling mongodb we did find a bug in openssl which would output the same error message, but this bug was fixed in June 2014, and the version of openssl we're using already has the fix applied did anyone else have this problem occur to them? is it an openssl bug, or mongodb bug? i've attached a log file of one of the affected servers, with debug mode enabled |
| Comments |
| Comment by Githook User [ 17/Apr/15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Author: {u'username': u'spencerjackson', u'name': u'Spencer Jackson', u'email': u'spencer.jackson@mongodb.com'}Message: (cherry picked from commit 83ae47b5780cddca30fd09b40fa4d897895a595f) | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Githook User [ 13/Mar/15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Author: {u'username': u'spencerjackson', u'name': u'Spencer Jackson', u'email': u'spencer.jackson@mongodb.com'}Message: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Githook User [ 13/Mar/15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Author: {u'username': u'spencerjackson', u'name': u'Spencer Jackson', u'email': u'spencer.jackson@mongodb.com'}Message: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 02/Mar/15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
james.wahlin@10gen.com tells me peter.garafano has a reproducer for this ticket, so I'm reopening it. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 02/Mar/15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
pcmaniac, we haven't heard back from you for a while so I'm resolving this ticket. If you find more time to investigate and have more information for us please feel free to re-open the ticket. Regards, | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Andreas Nilsson [ 07/Jan/15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi pcmaniac did you find out anything more? | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Nemanja Dubravac [ 15/Dec/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Ljuba tested with 2.8.0-rc0, but now when i check this version's changelog (https://jira.mongodb.org/issues/?jql=project%20%3D%20SERVER%20AND%20fixVersion%20%3D%20%222.8.0-rc0%22%20ORDER%20BY%20updated%20DESC%2C%20priority%20DESC%2C%20created%20ASC), i don't see that you're using versions >= 2.6.6, in which SSLv3 is disabled ( | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Andreas Nilsson [ 09/Dec/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Just a short update. We have set up several initial sync sessions over SSL with > 100GB without being able to regenerate the crash. Now we are trying to exactly mimic your setup in order to reproduce. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Andreas Nilsson [ 01/Dec/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thank you ljuba.ned@gmail.com. We will try to mimic your environment and get back to you with the results. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Ljuba Nedeljkovic [ 28/Nov/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Andreas, We've tried with 2.8.0 rc0 built with ssl support and same thing happens. @setup: you are correct. Below you can find output of rs.conf() for this minimal replica set as well as log excerpts upon mongod start.
Log exerpts: Primary:
Would-be secondary:
I hope this helps. Best, | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Andreas Nilsson [ 26/Nov/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I cannot come up with any explanation on top of my head. Maybe there is a resource exhaustion or something similar in OpenSSL that causes this for long-lived, data intensive sessions. If you have possibility to try it in your environment it would be interesting to see if the issue occurs in 2.8 RC0. We have disabled all SSLv3 ciphers due to POODLE, that might have a potential effects since the crash seems to occur in SSLv3 code. We're gonna try to reproduce this on our side with large data sets. Let me know if I understand the setup correctly:
Regards, | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Ljuba Nedeljkovic [ 24/Nov/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hello Andreas, Further experiments with initial sync of large database over SSL connection showed that:
I hope that this helps you to track down the issue. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Nemanja Dubravac [ 20/Nov/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
unfortunately, that's not the reason what mongodb lists in the log file is the output the same as the output of "openssl version" command: OpenSSL 1.0.1e-fips 11 Feb 2013 however, if i issue "rpm -q --changelog openssl", i see in the changelog that the latest change to openssl was made on "Thu Oct 16 2014", and the bugfix is included:
this serverfault page contains more information about how CentOS and RedHat issue package updates: http://serverfault.com/a/604277/102901 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Andreas Nilsson [ 17/Nov/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hey, I spent some time looking through the logs. I couldn't find anything conclusive right away. One thing I did note however is that the version of OpenSSL used by the server seems to be from Feb 2013.
so if there was a fix in June 2014 I guess it wouldn't be included. Can you double check the OpenSSL version on the target system. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Nemanja Dubravac [ 14/Nov/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi Andreas, Thanks for looking into this issue! in the log file I provided, there is first a part near one sample failure (2014-11-09T22:33:30.511+0100) then, there is a server restart with an emptied data directory, where you can see the whole process from initial startup, initial sync and including the first SSL exception (2014-11-10T10:54:30.514+0100)
the replica set is a part of a sharded cluster, but i don't think it makes a difference in this case we were able to do a resync after we stopped requiring ssl on this cluster, so it tells us that the problem is either in the openssl library, the way we compiled mongodb to use openssl, or in mongodb's implementation of communication over ssl thanks! | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Andreas Nilsson [ 14/Nov/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi pcmaniac, we will need some more information in order to reproduce and determine where the problem lies. It would be helpful to see more of the logs some time before the problem starts happening. Specifically it would be interesting to know details about the replica set and whether or not there were any stepdowns during the initial sync process. |