[SERVER-1704] mongod crashes, may be related to out of disk space Created: 31/Aug/10  Updated: 29/Aug/11  Resolved: 14/Jun/11

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 1.6.1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Alvin Richards (Inactive) Assignee: Eliot Horowitz (Inactive)
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

EC2 / Ubuntu 10.04


Operating System: Linux
Participants:

 Description   

Problem:
I was testing a data load script. During the testing I ran out of disk space.

From the mongo client, I could still query and get results

> db.blogs.count()
57389495
> db.blogs.find()

{ "_id" : ObjectId("4c7c20be8da65f35566b911f"), "author" : "jim", "ts" : "Mon Aug 30 2010 14:21:02 GMT-0700 (PDT)", "tags" : [ "yes", "no" ] } { "_id" : ObjectId("4c7c20be8da65f35566b9120"), "author" : "jim", "ts" : "Mon Aug 30 2010 14:21:02 GMT-0700 (PDT)", "tags" : [ "yes", "no" ] } { "_id" : ObjectId("4c7c20be8da65f35566b9121"), "author" : "jim", "ts" : "Mon Aug 30 2010 14:21:02 GMT-0700 (PDT)", "tags" : [ "yes", "no" ] }

However, after I performed a collection.drop() it appeared that the mongod process died

> db.blogs.drop()
true
> use scaleout
switched to db scaleout
> for (i=0; i < 10; i++) { db.blogs.insert(

{ author: 'jim', ts : new Date(), tags : ["yes","no"] }

); }
dbconnectionpool: connect failed 10.242.221.155:27000 : couldn't connect to server 10.242.221.155:27000}

Looking at the log file I see the following assert

Tue Aug 31 00:04:18 [conn15] Caught Assertion in insert , continuing
Tue Aug 31 00:04:18 [conn15] insert scaleout.blogs exception 0 assertion db/rec.h:126 22ms
Tue Aug 31 00:04:18 [conn3] end connection 10.242.221.155:51326
Tue Aug 31 00:04:18 [conn15] got request after shutdown()
Tue Aug 31 00:04:18 [conn1] end connection 127.0.0.1:38079
Tue Aug 31 00:04:18 [conn17] got request after shutdown()
Tue Aug 31 00:04:18 [conn16] got request after shutdown()
Tue Aug 31 00:04:18 [conn19] end connection 10.194.171.159:42089
Tue Aug 31 00:04:18 [conn18] end connection 10.194.171.159:42088
Tue Aug 31 00:04:18 [conn20] end connection 10.194.171.159:42090
Tue Aug 31 00:04:18 [initandlisten] now exiting
Tue Aug 31 00:04:18 dbexit: ; exiting immediately

Tue Aug 31 00:04:19 [conn4] end connection 10.194.102.4:44198
Tue Aug 31 00:04:20 Got signal: 11 (Segmentation fault).

Tue Aug 31 00:04:20 Backtrace:
0x813619 0x7f19d59d9af0 0x52a535 0x6f61fd 0x6f7b11 0x816220 0x829370 0x7f19d64dc9ca 0x7f19d5a8c69d
/var/opt/mongodb/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x813619]
/lib/libc.so.6(+0x33af0) [0x7f19d59d9af0]
/var/opt/mongodb/bin/mongod(_ZN5mongo9MongoFile13closeAllFilesERSt18basic_stringstreamIcSt11char_traitsIcESaIcEE+0xa5) [0x52a535]
/var/opt/mongodb/bin/mongod(_ZN5mongo8shutdownEv+0x39d) [0x6f61fd]
/var/opt/mongodb/bin/mongod(_ZN5mongo6dbexitENS_8ExitCodeEPKc+0x201) [0x6f7b11]
/var/opt/mongodb/bin/mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0x1060) [0x816220]
/var/opt/mongodb/bin/mongod(thread_proxy+0x80) [0x829370]
/lib/libpthread.so.0(+0x69ca) [0x7f19d64dc9ca]
/lib/libc.so.6(clone+0x6d) [0x7f19d5a8c69d]

Reproduce:

  • create a DB with limited free space (say 1GB)
  • insert into the DB and fill up the available disk space
  • once the DB is out of space, do a db.collection.drop()

Workaround:

  • allocate more disk space
  • restart the db with the "--repair" flag

Business Case:

  • Reliability
  • User Expereince


 Comments   
Comment by Alvin Richards (Inactive) [ 02/Mar/11 ]

Tried this again, its not reproducible. How do you want to proceed?

Comment by Aaron Staple [ 14/Feb/11 ]

no, it looks like InsertSpeed.java requires some command line arguments specifying what tests to run

Comment by Alvin Richards (Inactive) [ 14/Feb/11 ]

for starting up mongo?

Comment by Aaron Staple [ 14/Feb/11 ]

Do you know what command line options you used?

Comment by Aaron Staple [ 27/Oct/10 ]

Do you have the script that was used?

Comment by Alvin Richards (Inactive) [ 27/Oct/10 ]

Environment is no longer available, so these log are all I have.

Comment by Aaron Staple [ 27/Oct/10 ]

I wasn't able to reproduce this, I just got SERVER-2012.

From the log snippet it looks like several things are happening at once here - inserts, drops, shutdowns. Would be helpful if you could provide the full log. Is it still available?

Generated at Thu Feb 08 02:57:47 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.