Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-8783

Crash when out of space

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Minor - P4 Minor - P4
    • None
    • Affects Version/s: 2.0.6, 2.2.3
    • Component/s: None
    • None
    • Environment:
      Debian 6.0.3 x86_64
    • Linux
    • Hide

      On a 174G mongo db with less than 5G space left on disk:
      > db.runCommand(

      {compact: 'info'}

      )

      Wait.

      See it ont 2.0.6, upgrade to 2.2.3with, same reaction.

      It would be nice if the server cry about the error in the logs, stop adding things and continue to serve the clients instead of dying.

      Show
      On a 174G mongo db with less than 5G space left on disk: > db.runCommand( {compact: 'info'} ) Wait. See it ont 2.0.6, upgrade to 2.2.3with, same reaction. It would be nice if the server cry about the error in the logs, stop adding things and continue to serve the clients instead of dying.

      On a 174G mongo db with less than 5G space left on disk:
      > db.174GrunCommand(

      {compact: 'info'}

      )

      Thu Feb 28 15:04:57 [FileAllocator] allocating new datafile /srv/mongodb/logs.101, filling with zeroes...
      Thu Feb 28 15:04:57 [FileAllocator] FileAllocator: posix_fallocate failed: errno:28 No space left on device falling back
      Thu Feb 28 15:04:59 [FileAllocator] error: failed to allocate new file: /srv/mongodb/logs.101 size: 536608768 failure creating new datafile; lseek failed for fd 228 with errno: errno:28 No space left on device. will try again in 10 seconds
      Thu Feb 28 15:05:02 [initandlisten] connection accepted from 163.5.68.53:52434 #124 (117 connections now open)
      Thu Feb 28 15:05:03 [conn29] LogFile::synchronousAppend failed with 5300224 bytes unwritten out of 5300224 bytes; b=0x7f51129a8000 errno:28 No space left on device
      Thu Feb 28 15:05:03 [conn29] logs.info Fatal Assertion 13515
      0xb07561 0xacc8b3 0xae4b3f 0x73b215 0x73b442 0x73104b 0x73157a 0x72ef8c 0x72f09c 0x57cb06 0x57ecdd 0x57fb47 0x582cf0 0x6e8e51 0x6ea6c2 0x6eb67c 0x837468 0x83abb0 0x7b0afd 0x7b3e23
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xb07561]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xacc8b3]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo7LogFile17synchronousAppendEPKvm+0x14f) [0xae4b3f]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo3dur7Journal7journalERKNS0_11JSectHeaderERKNS_14AlignedBuilderE+0x1e5) [0x73b215]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo3dur14WRITETOJOURNALENS0_11JSectHeaderERNS_14AlignedBuilderE+0x32) [0x73b442]
      /usr/local/mongodb-current/bin/mongod() [0x73104b]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo3dur11DurableImpl9commitNowEv+0x1a) [0x73157a]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo3dur11DurableImpl16_aCommitIsNeededEv+0x14c) [0x72ef8c]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo3dur11DurableImpl14commitIfNeededEb+0x4c) [0x72f09c]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo13compactExtentEPKcPNS_16NamespaceDetailsENS_7DiskLocEiRKN5boost12scoped_arrayINS_9IndexSpecEEERNS6_INS_12SortPhaseOneEEEibdi+0x516) [0x57cb06]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo8_compactEPKcPNS_16NamespaceDetailsERSsbRNS_14BSONObjBuilderEdi+0x142d) [0x57ecdd]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo7compactERKSsRSsbRNS_14BSONObjBuilderEdi+0x277) [0x57fb47]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo10CompactCmd3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x950) [0x582cf0]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRNS_14BSONObjBuilderEb+0x51) [0x6e8e51]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xd52) [0x6ea6c2]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x2ac) [0x6eb67c]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x38) [0x837468]
      /usr/local/mongodb-current/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0xc10) [0x83abb0]
      /usr/local/mongodb-current/bin/mongod() [0x7b0afd]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x3a3) [0x7b3e23]
      Thu Feb 28 15:05:03 [conn29]

      ***aborting after fassert() failure

      Thu Feb 28 15:05:03 Got signal: 6 (Aborted).

      Thu Feb 28 15:05:03 Backtrace:
      0xb07561 0x5598c9 0x7fa31af36230 0x7fa31af361b5 0x7fa31af38fc0 0xacc8ee 0xae4b3f 0x73b215 0x73b442 0x73104b 0x73157a 0x72ef8c 0x72f09c 0x57cb06 0x57ecdd 0x57fb47 0x582cf0 0x6e8e51 0x6ea6c2 0x6eb67c
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xb07561]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x5598c9]
      /lib/libc.so.6(+0x32230) [0x7fa31af36230]
      /lib/libc.so.6(gsignal+0x35) [0x7fa31af361b5]
      /lib/libc.so.6(abort+0x180) [0x7fa31af38fc0]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo13fassertFailedEi+0xde) [0xacc8ee]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo7LogFile17synchronousAppendEPKvm+0x14f) [0xae4b3f]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo3dur7Journal7journalERKNS0_11JSectHeaderERKNS_14AlignedBuilderE+0x1e5) [0x73b215]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo3dur14WRITETOJOURNALENS0_11JSectHeaderERNS_14AlignedBuilderE+0x32) [0x73b442]
      /usr/local/mongodb-current/bin/mongod() [0x73104b]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo3dur11DurableImpl9commitNowEv+0x1a) [0x73157a]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo3dur11DurableImpl16_aCommitIsNeededEv+0x14c) [0x72ef8c]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo3dur11DurableImpl14commitIfNeededEb+0x4c) [0x72f09c]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo13compactExtentEPKcPNS_16NamespaceDetailsENS_7DiskLocEiRKN5boost12scoped_arrayINS_9IndexSpecEEERNS6_INS_12SortPhaseOneEEEibdi+0x516) [0x57cb06]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo8_compactEPKcPNS_16NamespaceDetailsERSsbRNS_14BSONObjBuilderEdi+0x142d) [0x57ecdd]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo7compactERKSsRSsbRNS_14BSONObjBuilderEdi+0x277) [0x57fb47]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo10CompactCmd3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x950) [0x582cf0]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRNS_14BSONObjBuilderEb+0x51) [0x6e8e51]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xd52) [0x6ea6c2]
      /usr/local/mongodb-current/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x2ac) [0x6eb67c]

            Assignee:
            Unassigned Unassigned
            Reporter:
            boudoi_b Benjamin Boudoir
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: