[SERVER-14238] Mongodb server crash after update of replica set member to 2.6 Created: 12/Jun/14 Updated: 10/Dec/14 Resolved: 30/Jun/14 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Storage |
| Affects Version/s: | 2.6.1 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Tudor Alexandru | Assignee: | Kaloian Manassiev |
| Resolution: | Duplicate | Votes: | 1 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||
| Operating System: | ALL | ||||||||||||
| Participants: | |||||||||||||
| Description |
|
As per http://docs.mongodb.org/manual/release-notes/2.6-upgrade/ i have followed the instructions for upgradming a replica set to 2.6. I have shut down the server with
After reading this |
| Comments |
| Comment by Ramon Fernandez Marina [ 30/Jun/14 ] | ||||||||||||||||||||||||||||||||||||||||
|
tudor_alexandru, thanks for the update. I'm convinced the root cause of the problem was duplicate indexes as described in Now that you're up and running again, you may want to consider validating your data to be absolutely sure there's no data corruption after the update (be aware that this is a very resource-intensive process and may have a performance impact on your MongoDB instance, so I'd recommend you run it on off-peak hours or, ideally, on a maintenance window). | ||||||||||||||||||||||||||||||||||||||||
| Comment by Tudor Alexandru [ 30/Jun/14 ] | ||||||||||||||||||||||||||||||||||||||||
|
@kaloian, i have managed to fix this in the following way: | ||||||||||||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 28/Jun/14 ] | ||||||||||||||||||||||||||||||||||||||||
|
kajsa_a, what you're seeing is tudor_alexandru, can you please check if you have any duplicate indexes? | ||||||||||||||||||||||||||||||||||||||||
| Comment by Kajsa Anderson [ 27/Jun/14 ] | ||||||||||||||||||||||||||||||||||||||||
|
I'm seeing the same error following upgrade from 2.4.4 to 2.6.3, though I see the database and collection name have been added to the error log. After downgrading back to 2.4.4, I'm able to see many duplicate indexes - that is, db.<collection>.getIndexes() shows several pairs like:
When I try to drop this index, either by name or by key pattern, I get:
| ||||||||||||||||||||||||||||||||||||||||
| Comment by Kaloian Manassiev [ 26/Jun/14 ] | ||||||||||||||||||||||||||||||||||||||||
|
Hi Tudor, The assertion, which you have hit most likely indicates that there are duplicate index entries in the namespace file (.ns file) of one of your databases. I would like to examine this file for possible inconsistencies and to get hints as to how this might have happened. Would it be possible that you share with us just the namespace file (NOT the data files themselves) ? The log does not indicate, which exactly database is corrupted, but hopefully you don't have too many databases. We could give you privately an scp share, where you can compress and securely upload the .ns file. Let me know if that works for you. -Kal. | ||||||||||||||||||||||||||||||||||||||||
| Comment by Tudor Alexandru [ 17/Jun/14 ] | ||||||||||||||||||||||||||||||||||||||||
|
I ran db.UpgradeCheckAllDBS over the weekend and it showed no errors. It has a very long output as it seems to check every document. | ||||||||||||||||||||||||||||||||||||||||
| Comment by Tudor Alexandru [ 13/Jun/14 ] | ||||||||||||||||||||||||||||||||||||||||
|
i shut down the replica with
and changed the binaries; i had the same result. | ||||||||||||||||||||||||||||||||||||||||
| Comment by Tudor Alexandru [ 12/Jun/14 ] | ||||||||||||||||||||||||||||||||||||||||
maybe we should have stopped the replica set with a different command?
if i run the command on the PRIMARY, it takes a lot of time.. | ||||||||||||||||||||||||||||||||||||||||
| Comment by Thomas Rueckstiess [ 12/Jun/14 ] | ||||||||||||||||||||||||||||||||||||||||
|
Hi Tudor, Thanks for reporting. To further investigate, could you upload the log file of the server that crashed reaching back to before you shut down the server on 2.4 for the upgrade? We'd like to see if the server shut down cleanly at that point. Are you also able to run the db.upgradeCheckAllDBs() shell helper to see if it picks up any issues? You don't need to upgrade the server again to run this check, just install a mongo shell version 2.6.1 somewhere (can be your laptop or another system) and connect to the server that crashed earlier. Then run:
and report back the output. Thanks, |