[SERVER-18448] mongod start failed Created: 13/May/15 Updated: 14/May/15 Resolved: 14/May/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | WiredTiger |
| Affects Version/s: | 3.0.2 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | naveen | Assignee: | Keith Bostic (Inactive) |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||
| Issue Links: |
|
||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||
| Operating System: | ALL | ||||||||
| Participants: | |||||||||
| Description |
|
I am using mongodb 3.0.2, storage engine - wiredTiger
when i running command for increasing hazard then the problem remain
|
| Comments |
| Comment by Ramon Fernandez Marina [ 14/May/15 ] | |||||||||||||||||||||||||||||||||||||||||
|
jdiego@digital-legends.com, please open a new SERVER ticket with details about your specific case. It could be that you case is also a duplicate of other issues already being investigated, but we'll need to investigate it first. Thanks, | |||||||||||||||||||||||||||||||||||||||||
| Comment by Juan Manuel Diego G [ 14/May/15 ] | |||||||||||||||||||||||||||||||||||||||||
|
keith.bostic How can I fix the files my self, I have a non replicated mongod instance, shutted it down with "mongod --shutdown" command and when trying to restart it failed. I'm on 3.0.3 by the way and "mongod --repair --dbpath /var/lib/mongodb --storageEngine wiredTiger" failed. | |||||||||||||||||||||||||||||||||||||||||
| Comment by Juan Manuel Diego G [ 14/May/15 ] | |||||||||||||||||||||||||||||||||||||||||
|
Ok, so I guess I'll have to remove all files of the dbPath to force full replica, do you know if the 3.0.3 has this fixed? because rackspace is in a reboot process right now, so it's possible some mongod processes will be terminated in a non elegant way. | |||||||||||||||||||||||||||||||||||||||||
| Comment by Keith Bostic (Inactive) [ 14/May/15 ] | |||||||||||||||||||||||||||||||||||||||||
|
jdiego@digital-legends.com, the fixed files were unique to naveendahiya1992's case. | |||||||||||||||||||||||||||||||||||||||||
| Comment by naveen [ 14/May/15 ] | |||||||||||||||||||||||||||||||||||||||||
|
Thanks for your suggestion, I will update mongodb as per stable version. | |||||||||||||||||||||||||||||||||||||||||
| Comment by Juan Manuel Diego G [ 14/May/15 ] | |||||||||||||||||||||||||||||||||||||||||
|
keith.bostic, does this fix applies for anyone? or the fixed files are unique to naveen's case? | |||||||||||||||||||||||||||||||||||||||||
| Comment by Keith Bostic (Inactive) [ 14/May/15 ] | |||||||||||||||||||||||||||||||||||||||||
|
naveendahiya1992, that's great to hear, thank you! We believe you ran into a recovery bug in WiredTiger, triggered by your power failure. The upcoming MongoDB releases will include fixes for this and a few related bugs, so we would strongly suggest upgrading to new stable releases of 3.0 as they come out. | |||||||||||||||||||||||||||||||||||||||||
| Comment by naveen [ 14/May/15 ] | |||||||||||||||||||||||||||||||||||||||||
|
may I know the problem behind the issue and what type of modifications you had done to those file ? | |||||||||||||||||||||||||||||||||||||||||
| Comment by naveen [ 14/May/15 ] | |||||||||||||||||||||||||||||||||||||||||
|
Thank you for your help. Data recovered successfully. | |||||||||||||||||||||||||||||||||||||||||
| Comment by naveen [ 14/May/15 ] | |||||||||||||||||||||||||||||||||||||||||
|
before corruption everything was running fine, I have been using this database from last 4 weeks and on May 12th my system was restarted due to power fluctuations at that time mongod was running and then when i tried to restart mongod using "service mongod restart" | |||||||||||||||||||||||||||||||||||||||||
| Comment by naveen [ 14/May/15 ] | |||||||||||||||||||||||||||||||||||||||||
|
Sorry, due to some problem I was unable to run it yesterday. 15-05-14T11:08:49.925+0530 I STORAGE [initandlisten] WiredTiger progress session.verify 1414280 , name: "id", ns: "TraiData.fulldata" } I will let you know when it will complete. | |||||||||||||||||||||||||||||||||||||||||
| Comment by Michael Cahill (Inactive) [ 14/May/15 ] | |||||||||||||||||||||||||||||||||||||||||
|
naveendahiya1992, is your system back up again? Do you need anything else from us? Can you share any information about what happened before the files got into this state? | |||||||||||||||||||||||||||||||||||||||||
| Comment by Keith Bostic (Inactive) [ 13/May/15 ] | |||||||||||||||||||||||||||||||||||||||||
|
naveendahiya1992, I've uploaded a second zip archive SERVER-18448-fixed-2 attachment that has repaired versions of both WiredTiger.wt and sizeStorer.wt. If you put those files into place, and then run normal mongod --repair, that will hopefully take care of the problem. Please let me know what happens, of course. | |||||||||||||||||||||||||||||||||||||||||
| Comment by naveen [ 13/May/15 ] | |||||||||||||||||||||||||||||||||||||||||
|
unable to repair, getting similar type of error:
| |||||||||||||||||||||||||||||||||||||||||
| Comment by Michael Cahill (Inactive) [ 13/May/15 ] | |||||||||||||||||||||||||||||||||||||||||
|
I think this should be handled by normal mongod --repair .... Can you please follow the instructions at http://docs.mongodb.org/manual/tutorial/recover-data-following-unexpected-shutdown/ ? | |||||||||||||||||||||||||||||||||||||||||
| Comment by naveen [ 13/May/15 ] | |||||||||||||||||||||||||||||||||||||||||
|
After changing permissions it executes and generate another error : file:sizeStorer.wt, session.open_cursor: read checksum error error log:
| |||||||||||||||||||||||||||||||||||||||||
| Comment by Michael Cahill (Inactive) [ 13/May/15 ] | |||||||||||||||||||||||||||||||||||||||||
|
That looks like a permissions issue: it might have been caused by decompressing and copying those files. Can you make sure that all the WiredTiger.* files in the database directory have the same owner, group and permissions? In other words, they look the same in ls -l apart from file size and date. | |||||||||||||||||||||||||||||||||||||||||
| Comment by naveen [ 13/May/15 ] | |||||||||||||||||||||||||||||||||||||||||
|
After replacing files error occured : Operation not permitted 2015-05-13T13:54:07.121+0530 I CONTROL ***** SERVER RESTARTED ***** | |||||||||||||||||||||||||||||||||||||||||
| Comment by Michael Cahill (Inactive) [ 13/May/15 ] | |||||||||||||||||||||||||||||||||||||||||
|
Can you try copying the contents of Please let me know whether this works, and I'm sorry for the inconvenience. Fixes for | |||||||||||||||||||||||||||||||||||||||||
| Comment by naveen [ 13/May/15 ] | |||||||||||||||||||||||||||||||||||||||||
|
These are all files except collection.wt and index.wt | |||||||||||||||||||||||||||||||||||||||||
| Comment by Michael Cahill (Inactive) [ 13/May/15 ] | |||||||||||||||||||||||||||||||||||||||||
|
Can you please upload all of the WiredTiger.* files from the database directory? Unfortunately, I can't give you a simple recipe to recover the WiredTiger.wt without a better understanding of what has gone wrong. | |||||||||||||||||||||||||||||||||||||||||
| Comment by naveen [ 13/May/15 ] | |||||||||||||||||||||||||||||||||||||||||
|
this is only copy of data. | |||||||||||||||||||||||||||||||||||||||||
| Comment by Michael Cahill (Inactive) [ 13/May/15 ] | |||||||||||||||||||||||||||||||||||||||||
|
This looks like What is your main goal: is this node part of a replica set or is the only copy of this data? |