Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-1704

mongod crashes, may be related to out of disk space

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Major - P3 Major - P3
    • None
    • 1.6.1
    • None
    • None
    • EC2 / Ubuntu 10.04
    • Linux

    Description

      Problem:
      I was testing a data load script. During the testing I ran out of disk space.

      From the mongo client, I could still query and get results

      > db.blogs.count()
      57389495
      > db.blogs.find()

      { "_id" : ObjectId("4c7c20be8da65f35566b911f"), "author" : "jim", "ts" : "Mon Aug 30 2010 14:21:02 GMT-0700 (PDT)", "tags" : [ "yes", "no" ] } { "_id" : ObjectId("4c7c20be8da65f35566b9120"), "author" : "jim", "ts" : "Mon Aug 30 2010 14:21:02 GMT-0700 (PDT)", "tags" : [ "yes", "no" ] } { "_id" : ObjectId("4c7c20be8da65f35566b9121"), "author" : "jim", "ts" : "Mon Aug 30 2010 14:21:02 GMT-0700 (PDT)", "tags" : [ "yes", "no" ] }

      However, after I performed a collection.drop() it appeared that the mongod process died

      > db.blogs.drop()
      true
      > use scaleout
      switched to db scaleout
      > for (i=0; i < 10; i++) { db.blogs.insert(

      { author: 'jim', ts : new Date(), tags : ["yes","no"] }

      ); }
      dbconnectionpool: connect failed 10.242.221.155:27000 : couldn't connect to server 10.242.221.155:27000}

      Looking at the log file I see the following assert

      Tue Aug 31 00:04:18 [conn15] Caught Assertion in insert , continuing
      Tue Aug 31 00:04:18 [conn15] insert scaleout.blogs exception 0 assertion db/rec.h:126 22ms
      Tue Aug 31 00:04:18 [conn3] end connection 10.242.221.155:51326
      Tue Aug 31 00:04:18 [conn15] got request after shutdown()
      Tue Aug 31 00:04:18 [conn1] end connection 127.0.0.1:38079
      Tue Aug 31 00:04:18 [conn17] got request after shutdown()
      Tue Aug 31 00:04:18 [conn16] got request after shutdown()
      Tue Aug 31 00:04:18 [conn19] end connection 10.194.171.159:42089
      Tue Aug 31 00:04:18 [conn18] end connection 10.194.171.159:42088
      Tue Aug 31 00:04:18 [conn20] end connection 10.194.171.159:42090
      Tue Aug 31 00:04:18 [initandlisten] now exiting
      Tue Aug 31 00:04:18 dbexit: ; exiting immediately

      Tue Aug 31 00:04:19 [conn4] end connection 10.194.102.4:44198
      Tue Aug 31 00:04:20 Got signal: 11 (Segmentation fault).

      Tue Aug 31 00:04:20 Backtrace:
      0x813619 0x7f19d59d9af0 0x52a535 0x6f61fd 0x6f7b11 0x816220 0x829370 0x7f19d64dc9ca 0x7f19d5a8c69d
      /var/opt/mongodb/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x813619]
      /lib/libc.so.6(+0x33af0) [0x7f19d59d9af0]
      /var/opt/mongodb/bin/mongod(_ZN5mongo9MongoFile13closeAllFilesERSt18basic_stringstreamIcSt11char_traitsIcESaIcEE+0xa5) [0x52a535]
      /var/opt/mongodb/bin/mongod(_ZN5mongo8shutdownEv+0x39d) [0x6f61fd]
      /var/opt/mongodb/bin/mongod(_ZN5mongo6dbexitENS_8ExitCodeEPKc+0x201) [0x6f7b11]
      /var/opt/mongodb/bin/mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0x1060) [0x816220]
      /var/opt/mongodb/bin/mongod(thread_proxy+0x80) [0x829370]
      /lib/libpthread.so.0(+0x69ca) [0x7f19d64dc9ca]
      /lib/libc.so.6(clone+0x6d) [0x7f19d5a8c69d]

      Reproduce:

      • create a DB with limited free space (say 1GB)
      • insert into the DB and fill up the available disk space
      • once the DB is out of space, do a db.collection.drop()

      Workaround:

      • allocate more disk space
      • restart the db with the "--repair" flag

      Business Case:

      • Reliability
      • User Expereince

      Attachments

        Activity

          People

            eliot Eliot Horowitz (Inactive)
            alvin Alvin Richards (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: