[SERVER-12650] Segfault when replicating from 2.4.6 Created: 06/Feb/14 Updated: 11/Jul/16 Resolved: 18/Feb/14 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication |
| Affects Version/s: | 2.5.5 |
| Fix Version/s: | 2.6.0-rc0 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Bob Potter | Assignee: | Matt Dannenberg |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Backwards Compatibility: | Fully Compatible |
| Operating System: | ALL |
| Steps To Reproduce: | Setup a 2.5.5 server as a replica of a 2.4.6 server. |
| Participants: |
| Description |
|
I'm running 2.5.5 as a hidden replica in a replSet that is composed of 2.4.6 instances. About once a day the server segfaults. I've included the backtrace and some of the log prior to the segfault. Let me know if I there is any additional information I can provide that would be useful.
There were also some interesting logs prior to the segfault:
|
| Comments |
| Comment by Githook User [ 20/Feb/14 ] | |||||||||||||||||||||||||||||||||||||||||
|
Author: {u'username': u'dannenberg', u'name': u'matt dannenberg', u'email': u'matt.dannenberg@10gen.com'}Message: | |||||||||||||||||||||||||||||||||||||||||
| Comment by Githook User [ 20/Feb/14 ] | |||||||||||||||||||||||||||||||||||||||||
|
Author: {u'username': u'dannenberg', u'name': u'matt dannenberg', u'email': u'matt.dannenberg@10gen.com'}Message: Revert " This reverts commit 617b4896241649e2a3203e33e5901907790002b5. | |||||||||||||||||||||||||||||||||||||||||
| Comment by Githook User [ 18/Feb/14 ] | |||||||||||||||||||||||||||||||||||||||||
|
Author: {u'username': u'dannenberg', u'name': u'Matt Dannenberg', u'email': u'matt.dannenberg@10gen.com'}Message: | |||||||||||||||||||||||||||||||||||||||||
| Comment by Bob Potter [ 13/Feb/14 ] | |||||||||||||||||||||||||||||||||||||||||
|
This is a production environment so I'm not currently able to update the primary to 2.4.9. I've updated the 2.5.5 replica to log level 3 and can provide you with better logs if it happens again. I saw the crash a couple times before I created the issue but haven't seen it since. | |||||||||||||||||||||||||||||||||||||||||
| Comment by Asya Kamsky [ 09/Feb/14 ] | |||||||||||||||||||||||||||||||||||||||||
|
I've replicated this failure on my mac (twice). Running 2.5.5 against 2.4.8 primary. The stack trace looks a little different but only in the system library files...
Logs from the 2.5.5 secondary (it was redoing full sync because that's what it died in the middle of the time before (when I had it in logLevel 1)
For completeness, log from the first crash ended on:
| |||||||||||||||||||||||||||||||||||||||||
| Comment by Matt Dannenberg [ 07/Feb/14 ] | |||||||||||||||||||||||||||||||||||||||||
|
We did some preliminary research and are struggling to find the path it's taking. If you could post longer logs from the crashing node with log verbosity set to 1 or greater, we could use the increased output for clues as to what's happening here. | |||||||||||||||||||||||||||||||||||||||||
| Comment by Scott Hernandez (Inactive) [ 06/Feb/14 ] | |||||||||||||||||||||||||||||||||||||||||
|
Can you please upload the full logs from the node that is crashing (with a few minutes before), as well as the primary at the time of the crash? If this happens regularly it would also be helpful if you can increase the verbosity of the logs (to level 3) before it happens: http://docs.mongodb.org/manual/reference/parameters/#param.logLevel Are you testing with the 2.5.5 development release in your production environment, or is this a test environment? If it is a test environment, can you please upgrade to the latest 2.4.x release (currently 2.4.9) to see if the problem still happens? It is important that we be able to see which member are communicating to which other members, so if you do replace any hostnames in the logs please make sure to be consistent with the replacement so we can still track the relationships. If you want to provide the logs privately let us know and we can create a (community) private issues which (will not be public) to post the logs to. |