[SERVER-8833] Segmentation fault after replSet relinquishing primary state, network/SSL related Created: 03/Mar/13 Updated: 08/Mar/13 Resolved: 04/Mar/13 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Stability |
| Affects Version/s: | 2.2.2 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Dave Claussen | Assignee: | Eric Milkie |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | connection, segfault, ssl | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Mongo is using SSL, in a clustered environment, with one primary, one secondary, and one arbiter. |
||
| Issue Links: |
|
||||||||
| Operating System: | Linux | ||||||||
| Steps To Reproduce: | 1. Set up a SSL mongo instnace with primary, secondary, and arbiter with host names (maybe in /etc/hosts). Note: I'm obviously guessing that the hostname issue is the cause, perhaps it's something completely different. But normally we run without issues for days on end without this happening. |
||||||||
| Participants: | |||||||||
| Description |
|
We use Cloudflare for our DNS. Their DNS went down this morning at 9:47 UTC as described here (http://techcrunch.com/2013/03/03/cloudflare-is-down-due-to-dns-outage-taking-down-785000-websites-including-4chan-wikileaks-metallica-com/). That caused our primary instance of MongoDB relinquishing primary state because we currently use hostnames in our replica set [I'll change this in the future so it does not]. That I would expect. But it also seg faulted: (I've changed the host names to be more descriptive) Sun Mar 3 03:52:12 [rsHealthPoll] getaddrinfo("secondary.ourhost.com") failed: Name or service not known Sun Mar 3 03:52:31 Got signal: 11 (Segmentation fault). Sun Mar 3 03:52:31 Backtrace: |
| Comments |
| Comment by Eric Milkie [ 04/Mar/13 ] | |||
|
Hi Dave. | |||
| Comment by Dave Claussen [ 03/Mar/13 ] | |||
|
FYI: Instead of changing the configuration to use IP addresses, I instead just put the hostnames into /etc/hosts. They don't change frequently, and I don't want primary to ever relinquish control because of a DNS issue, so this was the quickest (short-term) thing for me to do. | |||
| Comment by Dave Claussen [ 03/Mar/13 ] | |||
(Same as I get with yum info openssl-devel) | |||
| Comment by Scott Hernandez (Inactive) [ 03/Mar/13 ] | |||
|
What version of ssl do you have installed? I'm not that much of a centos expert but I think you can find out by running "yum info openssl-devel" or just doing a search for the libssl .so files (ls -laht /usr/lib/libssl*) (I updated the affects version) | |||
| Comment by Dave Claussen [ 03/Mar/13 ] | |||
|
Mongo DB version:
Note: My "affects version" in the original should be 2.2.2, not 2.2.0, that's a typo. It was built from source: | |||
| Comment by Scott Hernandez (Inactive) [ 03/Mar/13 ] | |||
|
What specific version of mongodb are you using (please post the exact "mongod --version" info), and how was it build and/or deployed (from source, a package, a tgz)? Please also attach the rest of your logs starting before the resolution errors. As an aside: generally you will want to stick with dns/host-names but setup caching on each host to deal with temporarily upstream resolution issues. |