[SERVER-8783] Crash when out of space Created: 28/Feb/13  Updated: 28/Feb/13  Resolved: 28/Feb/13

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 2.0.6, 2.2.3
Fix Version/s: None

Type: Bug Priority: Minor - P4
Reporter: Benjamin Boudoir Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Debian 6.0.3 x86_64


Issue Links:
Duplicate
duplicates SERVER-3759 filesystem ops may cause termination ... Closed
Operating System: Linux
Steps To Reproduce:

On a 174G mongo db with less than 5G space left on disk:
> db.runCommand(

{compact: 'info'}

)

Wait.

See it ont 2.0.6, upgrade to 2.2.3with, same reaction.

It would be nice if the server cry about the error in the logs, stop adding things and continue to serve the clients instead of dying.

Participants:

 Description   

On a 174G mongo db with less than 5G space left on disk:
> db.174GrunCommand(

{compact: 'info'}

)

Thu Feb 28 15:04:57 [FileAllocator] allocating new datafile /srv/mongodb/logs.101, filling with zeroes...
Thu Feb 28 15:04:57 [FileAllocator] FileAllocator: posix_fallocate failed: errno:28 No space left on device falling back
Thu Feb 28 15:04:59 [FileAllocator] error: failed to allocate new file: /srv/mongodb/logs.101 size: 536608768 failure creating new datafile; lseek failed for fd 228 with errno: errno:28 No space left on device. will try again in 10 seconds
Thu Feb 28 15:05:02 [initandlisten] connection accepted from 163.5.68.53:52434 #124 (117 connections now open)
Thu Feb 28 15:05:03 [conn29] LogFile::synchronousAppend failed with 5300224 bytes unwritten out of 5300224 bytes; b=0x7f51129a8000 errno:28 No space left on device
Thu Feb 28 15:05:03 [conn29] logs.info Fatal Assertion 13515
0xb07561 0xacc8b3 0xae4b3f 0x73b215 0x73b442 0x73104b 0x73157a 0x72ef8c 0x72f09c 0x57cb06 0x57ecdd 0x57fb47 0x582cf0 0x6e8e51 0x6ea6c2 0x6eb67c 0x837468 0x83abb0 0x7b0afd 0x7b3e23
/usr/local/mongodb-current/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xb07561]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xacc8b3]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo7LogFile17synchronousAppendEPKvm+0x14f) [0xae4b3f]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo3dur7Journal7journalERKNS0_11JSectHeaderERKNS_14AlignedBuilderE+0x1e5) [0x73b215]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo3dur14WRITETOJOURNALENS0_11JSectHeaderERNS_14AlignedBuilderE+0x32) [0x73b442]
/usr/local/mongodb-current/bin/mongod() [0x73104b]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo3dur11DurableImpl9commitNowEv+0x1a) [0x73157a]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo3dur11DurableImpl16_aCommitIsNeededEv+0x14c) [0x72ef8c]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo3dur11DurableImpl14commitIfNeededEb+0x4c) [0x72f09c]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo13compactExtentEPKcPNS_16NamespaceDetailsENS_7DiskLocEiRKN5boost12scoped_arrayINS_9IndexSpecEEERNS6_INS_12SortPhaseOneEEEibdi+0x516) [0x57cb06]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo8_compactEPKcPNS_16NamespaceDetailsERSsbRNS_14BSONObjBuilderEdi+0x142d) [0x57ecdd]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo7compactERKSsRSsbRNS_14BSONObjBuilderEdi+0x277) [0x57fb47]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo10CompactCmd3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x950) [0x582cf0]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRNS_14BSONObjBuilderEb+0x51) [0x6e8e51]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xd52) [0x6ea6c2]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x2ac) [0x6eb67c]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x38) [0x837468]
/usr/local/mongodb-current/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0xc10) [0x83abb0]
/usr/local/mongodb-current/bin/mongod() [0x7b0afd]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x3a3) [0x7b3e23]
Thu Feb 28 15:05:03 [conn29]

***aborting after fassert() failure

Thu Feb 28 15:05:03 Got signal: 6 (Aborted).

Thu Feb 28 15:05:03 Backtrace:
0xb07561 0x5598c9 0x7fa31af36230 0x7fa31af361b5 0x7fa31af38fc0 0xacc8ee 0xae4b3f 0x73b215 0x73b442 0x73104b 0x73157a 0x72ef8c 0x72f09c 0x57cb06 0x57ecdd 0x57fb47 0x582cf0 0x6e8e51 0x6ea6c2 0x6eb67c
/usr/local/mongodb-current/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xb07561]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x5598c9]
/lib/libc.so.6(+0x32230) [0x7fa31af36230]
/lib/libc.so.6(gsignal+0x35) [0x7fa31af361b5]
/lib/libc.so.6(abort+0x180) [0x7fa31af38fc0]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo13fassertFailedEi+0xde) [0xacc8ee]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo7LogFile17synchronousAppendEPKvm+0x14f) [0xae4b3f]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo3dur7Journal7journalERKNS0_11JSectHeaderERKNS_14AlignedBuilderE+0x1e5) [0x73b215]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo3dur14WRITETOJOURNALENS0_11JSectHeaderERNS_14AlignedBuilderE+0x32) [0x73b442]
/usr/local/mongodb-current/bin/mongod() [0x73104b]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo3dur11DurableImpl9commitNowEv+0x1a) [0x73157a]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo3dur11DurableImpl16_aCommitIsNeededEv+0x14c) [0x72ef8c]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo3dur11DurableImpl14commitIfNeededEb+0x4c) [0x72f09c]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo13compactExtentEPKcPNS_16NamespaceDetailsENS_7DiskLocEiRKN5boost12scoped_arrayINS_9IndexSpecEEERNS6_INS_12SortPhaseOneEEEibdi+0x516) [0x57cb06]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo8_compactEPKcPNS_16NamespaceDetailsERSsbRNS_14BSONObjBuilderEdi+0x142d) [0x57ecdd]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo7compactERKSsRSsbRNS_14BSONObjBuilderEdi+0x277) [0x57fb47]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo10CompactCmd3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x950) [0x582cf0]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRNS_14BSONObjBuilderEb+0x51) [0x6e8e51]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xd52) [0x6ea6c2]
/usr/local/mongodb-current/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x2ac) [0x6eb67c]


Generated at Thu Feb 08 03:18:26 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.