[SERVER-8007] mongod segfaulted Created: 21/Dec/12  Updated: 08/Apr/13  Resolved: 08/Apr/13

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 2.2.2
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Alister Burrell Assignee: Barrie Segal
Resolution: Done Votes: 0
Labels: crash, mongod, nh-240, segfault
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

CentOS release 6.3 (Final) running in AWS


Attachments: File mongod.log    
Operating System: Linux
Steps To Reproduce:

I would expect a repeat of the events in the mongod.log would re create this issue.

Participants:

 Description   

mongod segfaults as seen in the log snippet:

Dec 19 14:26:49 abrt[3665]: Saved core dump of pid 24104 (/usr/bin/mongod) to /var/spool/abrt/ccpp-2012-12-19-14:26:38-24104 (1784872960 bytes)
Dec 19 14:26:49 abrtd: Directory 'ccpp-2012-12-19-14:26:38-24104' creation detected
Dec 19 14:26:50 abrtd: Package 'mongo-10gen-server' isn't signed with proper key
Dec 19 14:26:50 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2012-12-19-14:26:38-24104' exited with 1
Dec 19 14:26:50 abrtd: Corrupted or bad directory /var/spool/abrt/ccpp-2012-12-19-14:26:38-24104, deleting

Around this time a number of issues and errors are seen in mongod.log which should have recovered and could be caused by possible issues in AWS or mongod itself we are currently unsure so hints here would also be great.



 Comments   
Comment by Alister Burrell [ 08/Apr/13 ]

Barrie,

We haven't managed to replicatate this since the first occurance even with the same settings as we had previously. I think you can probably close but if we see this again we will take full logs and re open a case.

Comment by Barrie Segal [ 05/Apr/13 ]

Hi Alister,

Are you still running into this issue?

Barrie

Comment by Barrie Segal [ 27/Mar/13 ]

Alister,

It looks like your open files and processes limits are too low. We recommend open files to be 64000 and processes to be 32000.

These recommendations (and others) can be found in our documentation here.

Barrie

Comment by Alister Burrell [ 11/Feb/13 ]

Our config hosts:

Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 10485760 unlimited bytes
Max core file size 0 0 bytes
Max resident set unlimited unlimited bytes
Max processes 1024 unlimited processes
Max open files 12000 12000 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 12868 12868 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

Our shards:

Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 10485760 unlimited bytes
Max core file size 0 0 bytes
Max resident set unlimited unlimited bytes
Max processes 1024 unlimited processes
Max open files 12000 12000 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 59359 59359 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

Comment by Barrie Segal [ 08/Feb/13 ]

Hi Alister,

Can you paste the output of:

$ cat /proc/<pid>/limits

substituting in the mongodb process ID?

Thanks,

Barrie

Comment by Alister Burrell [ 15/Jan/13 ]

Looks like the init script sets ulimit to 12000, do we want to increase it further?

Comment by Barrie Segal [ 04/Jan/13 ]

Alister,

Did you try increasing the ulimit?

Barrie

Comment by Eliot Horowitz (Inactive) [ 22/Dec/12 ]

Looks like the ulimit is too lower (default of 1024).
Can you try increasing that?
See: http://docs.mongodb.org/manual/administration/ulimit/

Generated at Thu Feb 08 03:16:17 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.