[SERVER-5808] init: mongodb main process terminated with status 100 when space is running low Created: 10/May/12  Updated: 11/Apr/18  Resolved: 11/Jan/13

Status: Closed
Project: Core Server
Component/s: Stability
Affects Version/s: 2.0.4
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Chris Manson Assignee: Adam Comerford
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu 11.10, 10 gen ubuntu packages using upstart, Virtual Machine using Virtual Box on Mac


Operating System: Linux
Participants:

 Description   

I was running an update command on a capped collection and the server suddenly started showing the following message:

Thu May 10 12:47:07 DBClientCursor::init call() failed
Thu May 10 12:47:07 query failed : ***** to: localhost
Error: error doing query: failed
Thu May 10 12:47:07 trying reconnect to localhost
Thu May 10 12:47:07 reconnect localhost ok
Thu May 10 12:47:07 DBClientCursor::init call() failed

I couldn't find anything in the logs but I happened to check dmesg and it had the entry

[ 49.842846] init: mongodb main process (1074) terminated with status 100

And when I tried to start mongo again I got another entry in dmesg

[ 225.928777] init: mongodb main process (1092) terminated with status 100

The only thing that I can think of being the problem is that the VM I'm using is almost out of space

=> / is using 96.3% of 1.70GB

I would imagine that if this is an issue with space then there should be something in the logs or something to point me in the right direction.



 Comments   
Comment by Christopher Reay [ 11/Apr/18 ]

Hello, I just came across this issue with 90% disk space used, and about 1.5GB of free diskspace on the server.

`service start mongod` reports back as "started, PID w/e"
`service --status-all`
mongod service reports running, where the mongodb service doesnt.

I found this because I couldnt connect to mongodb.
this happened after a restart of the machine (i.e. doesnt happen if the disk space is varying whilst the service is running, just when it is not).

I deleted a bunch of stuff, and am at about 70% disk usage with 5.5 GB free, and mongo starts fine
Cheers

Comment by Adam Comerford [ 17/Oct/12 ]

By my calculations, 96.3% of 1.7GB means there was just ~64MB left on the device here which is very low in real terms, if this was 1.7TB with 64GB to spare, I think it would be OK. The quickest way to determine if this was disk space would be to increase the size (it's a Virtual Box VM right?) and then see if the error still happens.

Adam.

Comment by Eliot Horowitz (Inactive) [ 12/May/12 ]

Is that from du?
When its that close, can be funny about allocations/fragmentation.

Comment by Chris Manson [ 11/May/12 ]

As I said in the the main description the disk is not full but it is 96% full

Comment by Eliot Horowitz (Inactive) [ 11/May/12 ]

Is the disk full?

Comment by Chris Manson [ 10/May/12 ]

Well there is noting in the log. It seems to have just ended a few days ago.

Sun Apr 29 06:50:00 [PeriodicTask::Runner] task: DBConnectionPool-cleaner took: 21ms
Sun Apr 29 06:50:48 [clientcursormon] mem (MB) res:1 virt:727 mapped:240
Sun Apr 29 06:51:00 [PeriodicTask::Runner] task: WriteBackManager::cleaner took: 6ms
Sun Apr 29 06:51:00 [PeriodicTask::Runner] task: DBConnectionPool-cleaner took: 4ms
Sun Apr 29 06:53:03 [conn2082] end connection 10.0.2.2:58158
Sun Apr 29 06:53:03 [initandlisten] connection accepted from 10.0.2.2:58220 #2086
Sun Apr 29 06:53:03 [conn2086] authenticate: ********
Sun Apr 29 06:53:03 [conn2086] command authentication.$cmd command: ******
Sun Apr 29 06:53:03 [conn2086] end connection 10.0.2.2:58220
Sun Apr 29 06:53:07 [initandlisten] connection accepted from 10.0.2.2:5822

I just did a ls -l on the log file after trying to start the mongodb service again and it seems to be last modified recently but nothing is getting appended

ubuntu@host-dev:~$ ls -l /var/log/mongodb/mongodb.log
-rw-r--r-- 1 mongodb nogroup 778240 2012-05-10 14:15 /var/log/mongodb/mongodb.log

Comment by Eliot Horowitz (Inactive) [ 10/May/12 ]

Can you send more of the log?

Generated at Thu Feb 08 03:09:56 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.