[SERVER-5242] mongod locks up on OSX after less than two days of uptime Created: 07/Mar/12  Updated: 15/Aug/12  Resolved: 11/Apr/12

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 2.0.3
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Arun Bhalla Assignee: Ben Becker
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

sys info: Darwin erh2.10gen.cc 9.8.0 Darwin Kernel Version 9.8.0: Wed Jul 15 16:55:01 PDT 2009; root:xnu-1228.15.4~1/RELEASE_I386 i386 BOOST_LIB_VERSION=1_40
OSX 10.6.8
2 x 2.66 GHz Dual-Core Intel Xeon
11 GB RAM


Attachments: HTML File mongod arun.staff.sf.atlassian.com.html    
Operating System: OS X
Participants:

 Description   

I installed mongodb 2.0.3 for OSX on Monday on my workstation for development use. It's been working fine until just a few minutes ago, when it refused new DB connections. I was running mongod in a separate terminal in non-daemon mode; the process wouldn't respond to Ctrl-C. Ultimately I had to kill -9 the process.

Console output before killing the process:

Wed Mar  7 12:17:23 [conn72] CMD: drop pac.tmp.mr.plugins_1790_inc
Wed Mar  7 12:17:23 [conn72] command pac.$cmd command: { mapreduce: "plugins", map: "
    function () {
      var pluginId = this._id;
      this.reviews.forEach(function(review) {
        emit(review._id, {pluginId: pluginId, review: ...", reduce: "
    function (key, values) {
      if (values.length > 0) return values[0];
    }", verbose: false, out: "reviews" } ntoreturn:1 reslen:131 275ms
got pipe signal:13
Wed Mar  7 12:19:37 [initandlisten] select() signal caught, continuing
got pipe signal:13
Wed Mar  7 12:19:37 [initandlisten] select() signal caught, continuing
got pipe signal:13
got pipe signal:13
got pipe signal:13
Wed Mar  7 12:19:37 [initandlisten] select() signal caught, continuing



 Comments   
Comment by Ben Becker [ 11/Apr/12 ]

Thanks Arun; please feel free to reopen this issue if you encounter this again.

Comment by Arun Bhalla [ 11/Apr/12 ]

No. The mongod process has been running for a month with no issue.

Comment by Ben Becker [ 11/Apr/12 ]

Hi Arun,

Have you run into any issues since increasing the file descriptor limit?

Comment by Ben Becker [ 12/Mar/12 ]

Thanks, please let me know if you run into this issue again. If everything works correctly with the increased limits, then this issue should be resolved by SERVER-4949.

Comment by Arun Bhalla [ 12/Mar/12 ]

OK, I've restarted mongod with a limit of 4096 files.

Comment by Ben Becker [ 10/Mar/12 ]

Hi Arun,

It looks like you're running out of file descriptors. The results of ulimit show a maximum of 256 open files and connections, which includes database files and socket connections. Could you try increasing the limit as mentioned in this wiki page?

Comment by Arun Bhalla [ 09/Mar/12 ]

Here's the ulimit -a output.

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 256
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 266
virtual memory          (kbytes, -v) unlimited

I have just a little bit more of the log output from Wednesday:

Wed Mar  7 12:17:23 [conn72] build index pac.tmp.mr.plugins_1790 { value.review.date: -1 }
Wed Mar  7 12:17:23 [conn72] build index done 0 records 0 secs
Wed Mar  7 12:17:23 [conn72] CMD: drop pac.reviews
Wed Mar  7 12:17:23 [conn72] CMD: drop pac.tmp.mr.plugins_1790
Wed Mar  7 12:17:23 [conn72] CMD: drop pac.tmp.mr.plugins_1790
Wed Mar  7 12:17:23 [conn72] CMD: drop pac.tmp.mr.plugins_1790_inc
Wed Mar  7 12:17:23 [conn72] command pac.$cmd command: { mapreduce: "plugins", map: "
    function () {
      var pluginId = this._id;
      this.reviews.forEach(function(review) {
        emit(review._id, {pluginId: pluginId, review: ...", reduce: "
    function (key, values) {
      if (values.length > 0) return values[0];
    }", verbose: false, out: "reviews" } ntoreturn:1 reslen:131 275ms
got pipe signal:13
Wed Mar  7 12:19:37 [initandlisten] select() signal caught, continuing
got pipe signal:13
Wed Mar  7 12:19:37 [initandlisten] select() signal caught, continuing
got pipe signal:13
got pipe signal:13
got pipe signal:13
Wed Mar  7 12:19:37 [initandlisten] select() signal caught, continuing

Let me know if you want me to run mongod any differently for debugging purposes, or if I should run any commands if this should happen again.

Comment by Ben Becker [ 09/Mar/12 ]

Hi Arun,

Could you run ulimit -a and paste the results? If mongod runs as a different user, please run this command as the other user (e.g. sudo su <username> then ulimit -a).

Do you still have the full log output from the console?

Generated at Thu Feb 08 03:08:18 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.