-
Type:
Bug
-
Resolution: Done
-
Priority:
Blocker - P1
-
None
-
Affects Version/s: 2.0.3
-
Component/s: Sharding
-
None
-
Environment:Linux (OEL)
-
ALL
-
None
-
0
-
None
-
None
-
None
-
None
-
None
-
None
Hi,
we got error in show dbs command:
mongos> show dbs
Wed Dec 12 16:45:53 uncaught exception: listDatabases failed:{
"assertion" : "runCommand (
) on shard (shard0000) failed :
{ assertion: \"can't map file memory\", assertionCode: 10085, errmsg: \"db assertion failure\", ok: 0.0 }",
"assertionCode" : 13136,
"errmsg" : "db assertion failure",
"ok" : 0
}
mongod logs:
Wed Dec 12 17:44:43 [conn5] command admin.$cmd command:
ntoreturn:1 reslen:67 0ms
Wed Dec 12 17:44:43 [conn785] runQuery called admin.$cmd
Wed Dec 12 17:44:43 [conn785] run command admin.$cmd
{ listDatabases: 1.0 }Wed Dec 12 17:44:43 [conn785] command:
{ listDatabases: 1.0 }Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00150612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00160612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00170612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00180612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00190612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00200612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00210612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00220612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00230612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00240612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00250612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00260612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00270612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00280612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00290612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00300612.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00010712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00020712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00030712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00040712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00050712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00060712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00070712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00080712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00090712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00100712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00110712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00120712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00130712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00140712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00150712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00160712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00170712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00180712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00190712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00200712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00210712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00220712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00230712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00240712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00250712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00260712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00270712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00280712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00290712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00300712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00310712.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00010812.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00020812.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00030812.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00040812.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00050812.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00060812.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00070812.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00080812.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00090812.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00100812.ns
Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00110812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00120812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00130812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00140812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00150812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00160812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00170812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00180812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00190812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00200812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00210812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00220812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00230812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00240812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00250812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00260812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00270812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00280812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00290812.ns
Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00300812.ns
Wed Dec 12 17:44:44 [conn785] Accessing: 00300812 for the first time
Wed Dec 12 17:44:44 [conn785] couldn't open /u01/shard1/00300812.10 errno:24 Too many open files
Wed Dec 12 17:44:44 [conn785] User Assertion: 10085:can't map file memory
Wed Dec 12 17:44:44 [conn785] command admin.$cmd command:
ntoreturn:1 exception: can't map file memory code:10085 reslen:126 101ms
Wed Dec 12 17:44:44 [conn785] runQuery called admin.$cmd
Wed Dec 12 17:44:44 [conn785] run command admin.$cmd
{ replSetGetStatus: 1, forShell: 1 }Wed Dec 12 17:44:44 [conn785] command:
{ replSetGetStatus: 1, forShell: 1 }Wed Dec 12 17:44:44 [conn785] command admin.$cmd command:
{ replSetGetStatus: 1, forShell: 1 } ntoreturn:1 reslen:76 0ms
Wed Dec 12 17:44:44 [conn3] insert 00121212.Database 0ms
Wed Dec 12 17:44:44 [conn3] runQuery called admin.$cmd
Wed Dec 12 17:44:44 [conn3] run command admin.$cmd
{ getlasterror: 1 }Wed Dec 12 17:44:44 [conn3] command admin.$cmd command:
{ getlasterror: 1 } ntoreturn:1 reslen:67 0ms
Wed Dec 12 17:44:44 [conn6] insert 00121212.Database 25ms
Wed Dec 12 17:44:44 [conn6] runQuery called admin.$cmd
Wed Dec 12 17:44:44 [conn6] run command admin.$cmd
{ getlasterror: 1 }Wed Dec 12 17:44:44 [conn6] command admin.$cmd command:
{ getlasterror: 1 } ntoreturn:1 reslen:67 0ms
Wed Dec 12 17:44:46 [conn785] SocketException: remote: 192.168.11.103:52451 error: 9001 socket exception [0] server [192.168.11.103:52451]
Wed Dec 12 17:44:47 [conn786] runQuery called admin.$cmd
Wed Dec 12 17:44:47 [conn786] run command admin.$cmd
{ whatsmyuri: 1 }Wed Dec 12 17:44:47 [conn786] command admin.$cmd command:
{ whatsmyuri: 1 } ntoreturn:1 reslen:67 0ms
Wed Dec 12 17:44:47 [conn786] runQuery called admin.$cmd
Wed Dec 12 17:44:47 [conn786] run command admin.$cmd
{ serverStatus: 1.0 }Wed Dec 12 17:44:47 [conn786] command admin.$cmd command:
{ serverStatus: 1.0 } ntoreturn:1 reslen:1363 0ms
Wed Dec 12 17:44:47 [conn786] SocketException: remote: 192.168.11.103:52570 error: 9001 socket exception [0] server [192.168.11.103:52570]
Wed Dec 12 17:44:47 [conn787] runQuery called admin.$cmd
Wed Dec 12 17:44:47 [conn787] run command admin.$cmd
{ whatsmyuri: 1 }Wed Dec 12 17:44:47 [conn787] command admin.$cmd command:
{ whatsmyuri: 1 } ntoreturn:1 reslen:67 0ms
Wed Dec 12 17:44:47 [conn787] runQuery called 00121212.$cmd { count: "Database", query: {}, fields: {} }
Wed Dec 12 17:44:47 [conn787] run command 00121212.$cmd { count: "Database", query: {}, fields: {} }
Wed Dec 12 17:44:47 [conn787] command 00121212.$cmd command: { count: "Database", query: {}, fields: {} } ntoreturn:1 reslen:48 0ms
Wed Dec 12 17:44:47 [conn787] SocketException: remote: 192.168.11.103:52586 error: 9001 socket exception [0] server [192.168.11.103:52586]
Wed Dec 12 17:44:48 [conn3] runQuery called admin.$cmd
According above logs file descriptor limit has been exhausted.
we have runned following command
1. System file descriptor limit :
sysctl fs.file-max
fs.file-max = 1601692
2. shell files descriptor limit :
ulimit -n
65535
3. Current files descriptor are in used
lsof | wc l
22845
I have following question.
1. What is the reason to come error "Too many file open":
2. Is there any think wrong?
please help me
regards,
JRV
- duplicates
-
SERVER-7022 File Descriptor Configuration for MongoDB Deployment
-
- Closed
-