[SERVER-14045] Cannot start mongodb Created: 25/May/14 Updated: 14/Nov/14 Resolved: 14/Nov/14 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Storage |
| Affects Version/s: | 2.6.1 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Blocker - P1 |
| Reporter: | Amin Cheloh | Assignee: | Ramon Fernandez Marina |
| Resolution: | Incomplete | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Linux node.xyz.com 3.11.0-20-generic #35~precise1-Ubuntu SMP Fri May 2 21:32:55 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
||
| Attachments: |
|
||||
| Issue Links: |
|
||||
| Operating System: | Linux | ||||
| Participants: | |||||
| Description |
|
After reboot server i cannot start mongodb server and i attach log file |
| Comments |
| Comment by Ramon Fernandez Marina [ 14/Nov/14 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
nineamin, since we haven't heard back from you for some time we're going to close this ticket. If this is still an issue for you feel free to reopen the ticket and provide additional details about your setup to see if you're affected by Regards, | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 17/Oct/14 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
nineamin, is this still an issue for you? If the answer is yes, have you checked | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 07/Oct/14 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
nineamin, are you running on VMWare by any chance? I'm asking because of | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Tyler Brock [ 06/Jun/14 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I'm going to mark this as resolved for the time being, let me know if you want us to dig further or you have any more questions. Thanks. | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Tyler Brock [ 04/Jun/14 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Amin thank you for sending us all of the files. Using a hex dump utility we were able to examine the "admin.ns" file which had several good entries and then what appears to be garbage written in the middle of the file. I've copied an series of excerpts from the hex dump below so you can see this for yourself:
This is likely to be a storage related issue as the damage starts on a 4kb boundary and the data doesn't look like anything mongod would have written to the namespace files. We can take a look at /var/log/messages, the syslog, and/or dmesg to look for msync and other storage layer related errors if you would like but it doesn't seem like this namespace file corruption was the result of code in mongod. Can you provide any other information about your virtualization setup? | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Amin Cheloh [ 01/Jun/14 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I don't know. No, It's impossible. Thanks | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Tyler Brock [ 30/May/14 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Amin, do you know why the log file shows many databases with other names such as "8111b8026c549b4c26b"? Is it possible that the actual data directory containing the vorchat database is somewhere other than the current dbpath mongo is using ("/var/lib/mongodb/")? | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Amin Cheloh [ 28/May/14 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
No, my database just only "vorchat". I also upload full /var/lib/mongodb/ backup (except journal directory) via scp Thanks | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Tyler Brock [ 27/May/14 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi Amin, Thanks for uploading the files. I see you uploaded a file called vorchat.zip. When I unzip that file I see a single database file (vorchat.0) and a namespace file (vorchat.ns). Is this related to the "8111b8026c549b4c26b" database involved in this ticket? | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Amin Cheloh [ 25/May/14 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
my workaround is delete admin.0 and admin.ns database files now i can start mongodb server and need recreate users. | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Amin Cheloh [ 25/May/14 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
the log is as attached file | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Eliot Horowitz (Inactive) [ 25/May/14 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Can you take a backup of the whole directory, and then remove the mongod.lock file and then try starting mongod | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Amin Cheloh [ 25/May/14 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
database files zip and attached additional information: Thanks | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Eliot Horowitz (Inactive) [ 25/May/14 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Do you know what collections if any should be in that database? | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Amin Cheloh [ 25/May/14 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
yes clean reboot result: If you are running a mongod on the same path you should connect to that instead of direct data file access 2014-05-25T10:36:28.140+0700 [tools] dbexit: so i try to run with --journal option and 1 directory, 0 files | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Eliot Horowitz (Inactive) [ 25/May/14 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
can you try running: Was it a clean reboot? |