[SERVER-26237] Mongodb failed to start after a disk 100% full problem Created: 22/Sep/16 Updated: 13/Aug/18 Resolved: 19/Nov/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | WiredTiger |
| Affects Version/s: | 3.2.3 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Paddy Zhu [X] | Assignee: | Ramon Fernandez Marina |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | envns, rdi, rps, trcf, wtc | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||
| Issue Links: |
|
||||
| Operating System: | Linux | ||||
| Participants: | |||||
| Description |
|
My server had a 'No space left on device' problem because our disk was running out of inode early today. But after we fixed the disk problem, our mongodb was still unable to start or repair. I desperately need your help : ( And the log information as follow:
|
| Comments |
| Comment by Kelsey Schubert [ 19/Nov/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi Paddyzhu, The likely cause of this issue was fixed in Kind regards, | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 03/Oct/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Apologies for the radio silence Paddyzhu. The goal behind looking at your data files was to see if this issue had corrupted them, or if only the repaired files were affected. If you haven't experienced any further issues and your data integrity hasn't been affected (you can run validate(), but note it's resource-intensive) then we don't need your files. Can you please confirm that you're not experiencing any issues that could be due to data corruption? If you are, then you should upload your database files. Thanks, | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Paddy Zhu [X] [ 23/Sep/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi everyone, We replaced our files in /usr/local/mongodb/data with your repair_attempt.zip and the issue is fixed! And since it is done do you still want our contents of /usr/local/mongodb/data ? If so I will upload it for you : ) | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Paddy Zhu [X] [ 23/Sep/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thank you! But It seems like I might just post the wrong log since I mixed other's with my own one. please check the following one and our mongodb.log I uploaded
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 22/Sep/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Paddyzhu, after further internal discussion we'd like to examine your database files to make sure you haven't found a new bug. Is it possible for you to upload the contents of /ick/data2 via this private, secure upload portal? Also, can you please upload the full log file for this node? I want to see if there are other clues before the failed restart that can help us determine what happened here. Thanks, | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 22/Sep/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Paddyzhu, I've run a repair attempt at the files you uploaded and attached them to this ticket. Please extract them in /ick/data2 and try to start mongod again, but note that there's no guarantee that this will work – I'd strongly recommend you use a replica set and keep good backups. I'd also recommend you upgrade to 3.2.9 (the latest 3.2 patch release at the time of this writing) as there have been many fixes to MongoDB 3.2 since 3.2.3. Thanks, |