[SERVER-18324] server crash on execute show dbs: too many open files Created: 05/May/15  Updated: 19/May/15  Resolved: 06/May/15

Status: Closed
Project: Core Server
Component/s: Admin
Affects Version/s: 3.0.2
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Guillermo [X] Assignee: Unassigned
Resolution: Done Votes: 0
Labels: centos, wiredtiger
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux CentOS 6.5
Linux localhost.localdomain 2.6.32-504.12.2.el6.x86_64 #1 SMP Wed Mar 11 22:03:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux


Attachments: Text File mongod.log    
Issue Links:
Related
Operating System: Linux
Participants:

 Description   

MongoDB shell version: 3.0.2
connecting to: test
Server has startup warnings:

2015-05-05T11:04:43.619+0200 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2015-05-05T11:04:43.619+0200 I CONTROL  [initandlisten]
2015-05-05T11:04:43.623+0200 I CONTROL  [initandlisten]
2015-05-05T11:04:43.623+0200 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2015-05-05T11:04:43.623+0200 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2015-05-05T11:04:43.623+0200 I CONTROL  [initandlisten]
2015-05-05T11:04:43.623+0200 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2015-05-05T11:04:43.623+0200 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2015-05-05T11:04:43.623+0200 I CONTROL  [initandlisten]

> show dbs
2015-05-05T11:05:35.371+0200 I NETWORK  DBClientCursor::init call() failed
2015-05-05T11:05:35.376+0200 E QUERY    Error: error doing query: failed
    at DBQuery._exec (src/mongo/shell/query.js:83:36)
    at DBQuery.hasNext (src/mongo/shell/query.js:240:10)
    at DBCollection.findOne (src/mongo/shell/collection.js:187:19)
    at DB.runCommand (src/mongo/shell/db.js:58:41)
    at Mongo.getDBs (src/mongo/shell/mongo.js:45:37)
    at shellHelper.show (src/mongo/shell/utils.js:630:33)
    at shellHelper (src/mongo/shell/utils.js:524:36)
    at (shellhelp2):1:1 at src/mongo/shell/query.js:83
2015-05-05T11:05:35.378+0200 I NETWORK  trying reconnect to 127.0.0.1:27017 (127.0.0.1) failed
2015-05-05T11:05:35.378+0200 W NETWORK  Failed to connect to 127.0.0.1:27017, reason: errno:111 Connection refused
2015-05-05T11:05:35.378+0200 I NETWORK  reconnect 127.0.0.1:27017 (127.0.0.1) failed failed couldn't connect to server 127.0.0.1:27017 (127.0.0.1), connection attempt failed
> uname
2015-05-05T11:13:29.511+0200 E QUERY    ReferenceError: uname is not defined
    at (shell):1:1



 Comments   
Comment by Ramon Fernandez Marina [ 07/May/15 ]

No worries GuillerVks, glad to hear you're able to continue testing MongoDB. Feel free to take a look at our Administration Refenrece for more information before you go into production.

Regards,
Ramón.

Comment by Guillermo [X] [ 06/May/15 ]

now works perfectly. i'm so stu***!, Yes, is a test instance, but thanks for your recommendation.
Thanks, sorry , and apologies the inconveniences.

Comment by Ramon Fernandez Marina [ 06/May/15 ]

GuillerVks, I'm able to reproduce the same behavior you observe and the same stack trace in the server logs when I lower the limit on the maximum number of files that can be opened. Please bump up this limit to allow for normal mongod operation. I'd also recommend you don't run mongod as root, as per the startup warning.

Regards,
Ramón.

Comment by Ramon Fernandez Marina [ 06/May/15 ]

Thanks GuillerVks. It seem your open files limit is too low (1024), and you have a few more files than that in your mongod data dir. Can you please change the open files limit to something larger like 4096 and try again?

Comment by Guillermo [X] [ 06/May/15 ]

Of Course.

show limits

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 14841
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Count number of files in /opt/mongo/data

1149

Regards.

Comment by Ramon Fernandez Marina [ 06/May/15 ]

Thanks for the additional information GuillerVks. Can you please also run the following commands from a shell and post the output?

# show limits
ulimit -a
# Count number of files in /opt/mongo/data
find /opt/mongo/data -type f | wc -l

Thanks,
Ramón.

Comment by Guillermo [X] [ 06/May/15 ]

Hi Ramon,
The server is running at moment that i execute the command 'show dbs', after the command, server shuts down. I Uploaded the server log. Regards.

Comment by Ramon Fernandez Marina [ 05/May/15 ]

GuillerVks, the error message indicates that mongod is no longer running. Can you check whether your mongod instance is running? Can you also upload the full server logs?

Generated at Thu Feb 08 03:47:18 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.