[SERVER-4038] mongod crashes preallocating next file when there's no more disk space available Created: 07/Oct/11 Updated: 25/Jun/15 Resolved: 24/Mar/14 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 2.0.0 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Daniel Pasette (Inactive) | Assignee: | Mathias Stearn |
| Resolution: | Done | Votes: | 1 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Operating System: | ALL | ||||||||
| Participants: | |||||||||
| Description |
|
I just got off a sales call with a prospect who is evaluating MongoDB. They claim that mongod crashes when it turns out of disk space. They Dwight sez: "i think that is correct. we have not done any work on out of disk space really." |
| Comments |
| Comment by Mathias Stearn [ 24/Mar/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||
|
This is intentional. The specific error is when writing to the journal. If there is no room to write the journal (or writing fails for any reason) we explicitly abort the server. The rational is that is is better to crash and leave the data in a consistent state than to potentially corrupt the data. | |||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Stennie Steneker (Inactive) [ 12/Oct/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||
|
There was a report of an ungraceful crash in 2.0.7 with stacktrace included from:
| |||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Xavier Tesch [ 24/Sep/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||
|
I believe there is a big issue there. Real world example: -A database with 10GB data size, 14GB storage size and mongodb files taking 30GB on disk because of big deletions. MongoDB behaviour: -Impossible to save anything in DB due to write lock because it cannot preallocate next file. IMHO mongod should never be in that situation where it is not functional despite having a LOT of space in already allocated files. This is a very annoying bug that should be fixed as soon as possible. The correct behaviour would be to avoid trying to preallocate when not enough disk space is available and not locking the writes when there is enough space in already existing files. | |||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Mathias Stearn [ 08/Oct/11 ] | |||||||||||||||||||||||||||||||||||||||||||||||
|
see: https://jira.mongodb.org/browse/SERVER-2609 Right now we prevent taking a write lock but still allow reads after a data-file prealloc fails. We then periodically retry to allocate the file to recover if the user makes more space available. A good enhancement would be to set the new maintnance mode to force a stepdown and make the node temporarily hidden. | |||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Dwight Merriman [ 08/Oct/11 ] | |||||||||||||||||||||||||||||||||||||||||||||||
|
Currently, it is supposed to terminate on out of disk space. Of course that may not be ideal. If it seg faults, that is definitely a bug and should be fixed right away. For the long term what is the right behavior:
|