Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-7911

Error "too many open file"

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Blocker - P1 Blocker - P1
    • None
    • Affects Version/s: 2.0.3
    • Component/s: Sharding
    • Labels:
      None
    • Environment:
      Linux (OEL)
    • ALL

      Hi,
      we got error in show dbs command:

      mongos> show dbs
      Wed Dec 12 16:45:53 uncaught exception: listDatabases failed:{
      "assertion" : "runCommand (

      { listDatabases: 1 }

      ) on shard (shard0000) failed :

      { assertion: \"can't map file memory\", assertionCode: 10085, errmsg: \"db assertion failure\", ok: 0.0 }

      ",
      "assertionCode" : 13136,
      "errmsg" : "db assertion failure",
      "ok" : 0
      }

      mongod logs:
      Wed Dec 12 17:44:43 [conn5] command admin.$cmd command:

      { getlasterror: 1 }

      ntoreturn:1 reslen:67 0ms
      Wed Dec 12 17:44:43 [conn785] runQuery called admin.$cmd

      { listDatabases: 1.0 }

      Wed Dec 12 17:44:43 [conn785] run command admin.$cmd

      { listDatabases: 1.0 }

      Wed Dec 12 17:44:43 [conn785] command:

      { listDatabases: 1.0 }

      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00150612.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00160612.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00170612.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00180612.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00190612.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00200612.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00210612.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00220612.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00230612.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00240612.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00250612.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00260612.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00270612.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00280612.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00290612.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00300612.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00010712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00020712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00030712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00040712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00050712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00060712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00070712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00080712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00090712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00100712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00110712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00120712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00130712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00140712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00150712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00160712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00170712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00180712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00190712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00200712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00210712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00220712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00230712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00240712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00250712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00260712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00270712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00280712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00290712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00300712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00310712.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00010812.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00020812.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00030812.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00040812.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00050812.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00060812.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00070812.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00080812.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00090812.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00100812.ns
      Wed Dec 12 17:44:43 [conn785] checking size file /u01/shard1/00110812.ns
      Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00120812.ns
      Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00130812.ns
      Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00140812.ns
      Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00150812.ns
      Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00160812.ns
      Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00170812.ns
      Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00180812.ns
      Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00190812.ns
      Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00200812.ns
      Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00210812.ns
      Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00220812.ns
      Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00230812.ns
      Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00240812.ns
      Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00250812.ns
      Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00260812.ns
      Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00270812.ns
      Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00280812.ns
      Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00290812.ns
      Wed Dec 12 17:44:44 [conn785] checking size file /u01/shard1/00300812.ns
      Wed Dec 12 17:44:44 [conn785] Accessing: 00300812 for the first time
      Wed Dec 12 17:44:44 [conn785] couldn't open /u01/shard1/00300812.10 errno:24 Too many open files
      Wed Dec 12 17:44:44 [conn785] User Assertion: 10085:can't map file memory
      Wed Dec 12 17:44:44 [conn785] command admin.$cmd command:

      { listDatabases: 1.0 }

      ntoreturn:1 exception: can't map file memory code:10085 reslen:126 101ms
      Wed Dec 12 17:44:44 [conn785] runQuery called admin.$cmd

      { replSetGetStatus: 1, forShell: 1 }

      Wed Dec 12 17:44:44 [conn785] run command admin.$cmd

      { replSetGetStatus: 1, forShell: 1 }

      Wed Dec 12 17:44:44 [conn785] command:

      { replSetGetStatus: 1, forShell: 1 }

      Wed Dec 12 17:44:44 [conn785] command admin.$cmd command:

      { replSetGetStatus: 1, forShell: 1 }

      ntoreturn:1 reslen:76 0ms
      Wed Dec 12 17:44:44 [conn3] insert 00121212.Database 0ms
      Wed Dec 12 17:44:44 [conn3] runQuery called admin.$cmd

      { getlasterror: 1 }

      Wed Dec 12 17:44:44 [conn3] run command admin.$cmd

      { getlasterror: 1 }

      Wed Dec 12 17:44:44 [conn3] command admin.$cmd command:

      { getlasterror: 1 }

      ntoreturn:1 reslen:67 0ms
      Wed Dec 12 17:44:44 [conn6] insert 00121212.Database 25ms
      Wed Dec 12 17:44:44 [conn6] runQuery called admin.$cmd

      { getlasterror: 1 }

      Wed Dec 12 17:44:44 [conn6] run command admin.$cmd

      { getlasterror: 1 }

      Wed Dec 12 17:44:44 [conn6] command admin.$cmd command:

      { getlasterror: 1 }

      ntoreturn:1 reslen:67 0ms
      Wed Dec 12 17:44:46 [conn785] SocketException: remote: 192.168.11.103:52451 error: 9001 socket exception [0] server [192.168.11.103:52451]
      Wed Dec 12 17:44:47 [conn786] runQuery called admin.$cmd

      { whatsmyuri: 1 }

      Wed Dec 12 17:44:47 [conn786] run command admin.$cmd

      { whatsmyuri: 1 }

      Wed Dec 12 17:44:47 [conn786] command admin.$cmd command:

      { whatsmyuri: 1 }

      ntoreturn:1 reslen:67 0ms
      Wed Dec 12 17:44:47 [conn786] runQuery called admin.$cmd

      { serverStatus: 1.0 }

      Wed Dec 12 17:44:47 [conn786] run command admin.$cmd

      { serverStatus: 1.0 }

      Wed Dec 12 17:44:47 [conn786] command admin.$cmd command:

      { serverStatus: 1.0 }

      ntoreturn:1 reslen:1363 0ms
      Wed Dec 12 17:44:47 [conn786] SocketException: remote: 192.168.11.103:52570 error: 9001 socket exception [0] server [192.168.11.103:52570]
      Wed Dec 12 17:44:47 [conn787] runQuery called admin.$cmd

      { whatsmyuri: 1 }

      Wed Dec 12 17:44:47 [conn787] run command admin.$cmd

      { whatsmyuri: 1 }

      Wed Dec 12 17:44:47 [conn787] command admin.$cmd command:

      { whatsmyuri: 1 }

      ntoreturn:1 reslen:67 0ms
      Wed Dec 12 17:44:47 [conn787] runQuery called 00121212.$cmd { count: "Database", query: {}, fields: {} }
      Wed Dec 12 17:44:47 [conn787] run command 00121212.$cmd { count: "Database", query: {}, fields: {} }
      Wed Dec 12 17:44:47 [conn787] command 00121212.$cmd command: { count: "Database", query: {}, fields: {} } ntoreturn:1 reslen:48 0ms
      Wed Dec 12 17:44:47 [conn787] SocketException: remote: 192.168.11.103:52586 error: 9001 socket exception [0] server [192.168.11.103:52586]
      Wed Dec 12 17:44:48 [conn3] runQuery called admin.$cmd

      { getlasterror: 1 }

      According above logs file descriptor limit has been exhausted.
      we have runned following command
      1. System file descriptor limit :
      sysctl fs.file-max
      fs.file-max = 1601692
      2. shell files descriptor limit :
      ulimit -n
      65535

      3. Current files descriptor are in used
      lsof | wc l
      22845

      I have following question.
      1. What is the reason to come error "Too many file open":
      2. Is there any think wrong?

      please help me

      regards,
      JRV

            Assignee:
            stephen.steneker@mongodb.com Stennie Steneker (Inactive)
            Reporter:
            jitendra908 jitendra
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: