Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-18960

confused log: pthread_create failed: errno:11 Resource temporarily unavailable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Done
    • Affects Version/s: 2.6.10
    • Fix Version/s: None
    • Component/s: Networking
    • Labels:
      None
    • Operating System:
      ALL

      Description

      There are some logs as below show up in mongod start progress:

      2015-06-13T04:08:01.238+0800 [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable
      2015-06-13T04:08:01.238+0800 [initandlisten] can't create new thread, closing connection
      ...
      2015-06-13T04:08:21.247+0800 [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable
      2015-06-13T04:08:21.248+0800 [initandlisten] can't create new thread, closing connection
      

      And the limits and threads count of mongod instance that I collected as below:

      2015-06-13_04:08:03
      limits:
      [mongod #28309 -- limits]
      Limit                     Soft Limit           Hard Limit           Units     
      Max cpu time              unlimited            unlimited            seconds   
      Max file size             unlimited            unlimited            bytes     
      Max data size             unlimited            unlimited            bytes     
      Max stack size            8388608              unlimited            bytes     
      Max core file size        0                    unlimited            bytes     
      Max resident set          unlimited            unlimited            bytes     
      Max processes             655350               655350               processes 
      Max open files            655350               655350               files     
      Max locked memory         65536                65536                bytes     
      Max address space         unlimited            unlimited            bytes     
      Max file locks            unlimited            unlimited            locks     
      Max pending signals       513987               513987               signals   
      Max msgqueue size         819200               819200               bytes     
      Max nice priority         0                    0                    
      Max realtime priority     0                    0                    
      Max realtime timeout      unlimited            unlimited            us        
      threads_count:
      [mongod #28309 -- limits]
      97
       
      limits:
      [mongod #28309 -- limits]
      Limit                     Soft Limit           Hard Limit           Units     
      Max cpu time              unlimited            unlimited            seconds   
      Max file size             unlimited            unlimited            bytes     
      Max data size             unlimited            unlimited            bytes     
      Max stack size            8388608              unlimited            bytes     
      Max core file size        0                    unlimited            bytes     
      Max resident set          unlimited            unlimited            bytes     
      Max processes             655350               655350               processes 
      Max open files            655350               655350               files     
      Max locked memory         65536                65536                bytes     
      Max address space         unlimited            unlimited            bytes     
      Max file locks            unlimited            unlimited            locks     
      Max pending signals       513987               513987               signals   
      Max msgqueue size         819200               819200               bytes     
      Max nice priority         0                    0                    
      Max realtime priority     0                    0                    
      Max realtime timeout      unlimited            unlimited            us        
      threads_count:
      [mongod #28309 -- limits]
      97
      

      I'm confused that the threads of mongod instance is less than 1000, and less than 655350, why log show that? Is it a bug?
      ps. This (RS) member have more than 10k databases and 700k collections( with indexes).
      ps. I use grab-mongodb-limits-and-threads.sh to get limits and threads count info above.

      grab-mongodb-limits-and-threads.sh

      #!/bin/bash
       
      return-limits(){
       
           for process in $@; do
                process_pids=`ps -C $process -o pid --no-headers | cut -d " " -f 2`
       
                if [ -z $@ ]; then
                   echo "[no $process running]"
                else
                   for pid in $process_pids; do
                         echo "[$process #$pid -- limits]"
                         cat /proc/$pid/limits
                   done
                fi
           done
      }
       
      return-tasks(){
       
           for process in $@; do
                process_pids=`ps -C $process -o pid --no-headers | cut -d " " -f 2`
       
                if [ -z $@ ]; then
                   echo "[no $process running]"
                else
                   for pid in $process_pids; do
                         echo "[$process #$pid -- limits]"
                         ls /proc/$pid/task | wc -l
                   done
                fi
           done
      }
       
      date +%F_%T
      echo "limits:"
      return-limits mongod
      echo "threads_count:"
      return-tasks mongod
      

        Attachments

          Activity

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: