[SERVER-7911] Error "too many open file" Created: 12/Dec/12  Updated: 15/Feb/13  Resolved: 14/Dec/12

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 2.0.3
Fix Version/s: None

Type: Bug Priority: Blocker - P1
Reporter: jitendra Assignee: Stennie Steneker (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux (OEL)


Issue Links:
Duplicate
duplicates SERVER-7022 File Descriptor Configuration for Mon... Closed
Operating System: ALL
Participants:

 Description   

Hi,
we got error in show dbs command:

mongos> show dbs
Wed Dec 12 16:45:53 uncaught exception: listDatabases failed:{
"assertion" : "runCommand (

{ listDatabases: 1 }

) on shard (shard0000) failed :

{ assertion: \"can't map file memory\", assertionCode: 10085, errmsg: \"db assertion failure\", ok: 0.0 }

",
"assertionCode" : 13136,
"errmsg" : "db assertion failure",
"ok" : 0
}

mongod logs:
Wed Dec 12 17:44:43 [conn5] command admin.$cmd command:

{ getlasterror: 1 }

ntoreturn:1 reslen:67 0ms
Wed Dec 12 17:44:43 [conn785] runQuery called admin.$cmd

{ listDatabases: 1.0 }

Wed Dec 12 17:44:43 [conn785] run command admin.$cmd

{ listDatabases: 1.0 }

Wed Dec 12 17:44:43 [conn785] command:

{ listDatabases: 1.0 }

Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00150612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00160612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00170612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00180612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00190612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00200612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00210612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00220612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00230612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00240612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00250612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00260612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00270612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00280612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00290612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00300612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00010712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00020712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00030712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00040712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00050712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00060712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00070712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00080712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00090712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00100712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00110712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00120712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00130712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00140712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00150712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00160712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00170712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00180712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00190712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00200712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00210712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00220712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00230712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00240712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00250712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00260712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00270712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00280712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00290712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00300712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00310712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00010812.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00020812.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00030812.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00040812.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00050812.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00060812.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00070812.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00080812.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00090812.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00100812.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00110812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00120812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00130812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00140812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00150812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00160812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00170812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00180812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00190812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00200812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00210812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00220812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00230812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00240812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00250812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00260812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00270812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00280812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00290812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00300812.ns
Wed Dec 12 17:44:44 [conn785] Accessing: 00300812 for the first time
Wed Dec 12 17:44:44 [conn785] couldn't open /u01/shard1/00300812.10 errno:24 Too many open files
Wed Dec 12 17:44:44 [conn785] User Assertion: 10085:can't map file memory
Wed Dec 12 17:44:44 [conn785] command admin.$cmd command:

{ listDatabases: 1.0 }

ntoreturn:1 exception: can't map file memory code:10085 reslen:126 101ms
Wed Dec 12 17:44:44 [conn785] runQuery called admin.$cmd

{ replSetGetStatus: 1, forShell: 1 }

Wed Dec 12 17:44:44 [conn785] run command admin.$cmd

{ replSetGetStatus: 1, forShell: 1 }

Wed Dec 12 17:44:44 [conn785] command:

{ replSetGetStatus: 1, forShell: 1 }

Wed Dec 12 17:44:44 [conn785] command admin.$cmd command:

{ replSetGetStatus: 1, forShell: 1 }

ntoreturn:1 reslen:76 0ms
Wed Dec 12 17:44:44 [conn3] insert 00121212.Database 0ms
Wed Dec 12 17:44:44 [conn3] runQuery called admin.$cmd

{ getlasterror: 1 }

Wed Dec 12 17:44:44 [conn3] run command admin.$cmd

{ getlasterror: 1 }

Wed Dec 12 17:44:44 [conn3] command admin.$cmd command:

{ getlasterror: 1 }

ntoreturn:1 reslen:67 0ms
Wed Dec 12 17:44:44 [conn6] insert 00121212.Database 25ms
Wed Dec 12 17:44:44 [conn6] runQuery called admin.$cmd

{ getlasterror: 1 }

Wed Dec 12 17:44:44 [conn6] run command admin.$cmd

{ getlasterror: 1 }

Wed Dec 12 17:44:44 [conn6] command admin.$cmd command:

{ getlasterror: 1 }

ntoreturn:1 reslen:67 0ms
Wed Dec 12 17:44:46 [conn785] SocketException: remote: 192.168.11.103:52451 error: 9001 socket exception [0] server [192.168.11.103:52451]
Wed Dec 12 17:44:47 [conn786] runQuery called admin.$cmd

{ whatsmyuri: 1 }

Wed Dec 12 17:44:47 [conn786] run command admin.$cmd

{ whatsmyuri: 1 }

Wed Dec 12 17:44:47 [conn786] command admin.$cmd command:

{ whatsmyuri: 1 }

ntoreturn:1 reslen:67 0ms
Wed Dec 12 17:44:47 [conn786] runQuery called admin.$cmd

{ serverStatus: 1.0 }

Wed Dec 12 17:44:47 [conn786] run command admin.$cmd

{ serverStatus: 1.0 }

Wed Dec 12 17:44:47 [conn786] command admin.$cmd command:

{ serverStatus: 1.0 }

ntoreturn:1 reslen:1363 0ms
Wed Dec 12 17:44:47 [conn786] SocketException: remote: 192.168.11.103:52570 error: 9001 socket exception [0] server [192.168.11.103:52570]
Wed Dec 12 17:44:47 [conn787] runQuery called admin.$cmd

{ whatsmyuri: 1 }

Wed Dec 12 17:44:47 [conn787] run command admin.$cmd

{ whatsmyuri: 1 }

Wed Dec 12 17:44:47 [conn787] command admin.$cmd command:

{ whatsmyuri: 1 }

ntoreturn:1 reslen:67 0ms
Wed Dec 12 17:44:47 [conn787] runQuery called 00121212.$cmd { count: "Database", query: {}, fields: {} }
Wed Dec 12 17:44:47 [conn787] run command 00121212.$cmd { count: "Database", query: {}, fields: {} }
Wed Dec 12 17:44:47 [conn787] command 00121212.$cmd command: { count: "Database", query: {}, fields: {} } ntoreturn:1 reslen:48 0ms
Wed Dec 12 17:44:47 [conn787] SocketException: remote: 192.168.11.103:52586 error: 9001 socket exception [0] server [192.168.11.103:52586]
Wed Dec 12 17:44:48 [conn3] runQuery called admin.$cmd

{ getlasterror: 1 }

According above logs file descriptor limit has been exhausted.
we have runned following command
1. System file descriptor limit :
sysctl fs.file-max
fs.file-max = 1601692
2. shell files descriptor limit :
ulimit -n
65535

3. Current files descriptor are in used
lsof | wc l
22845

I have following question.
1. What is the reason to come error "Too many file open":
2. Is there any think wrong?

please help me

regards,
JRV



 Comments   
Comment by Stennie Steneker (Inactive) [ 14/Dec/12 ]

Hi Jitendra,

The Core Server project is intended for filing feature requests and bug reports.

If you'd like someone to help troubleshoot an issue, please create a new discussion in the Google Group:
https://groups.google.com/forum/?fromgroups#!forum/mongodb-user

1. What is the reason to come error "Too many file open":

You have reached the limit of open files/connections and need to adjust your process and system ulimit settings appropriately:
http://docs.mongodb.org/manual/administration/ulimit/

2. Is there any think wrong?

Per the error message, your server configuration needs to open more files/connections than are available.

Based on your comment in SERVER-7022, you are running all of your sharded environment on a single machine and are also using --smallfiles which will create more data files on disk. You may be running into the system ulimits rather than process limits given all are running on the same machine.

In any case, the best forum for continued discussion would be the mongodb-user group.

Thanks,
Stephen

Generated at Thu Feb 08 03:15:56 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.