Major - P3 I use supervisor to start 200 python worker, each worker would import a MongoClient, I had tried `maxPoolSize=None` and `maxPoolSize=3000`, just same error.
I have check mongodb connection:
> db.serverStatus().connections
{ "current" : 1621, "available" : 18379, "totalCreated" : NumberLong(2512) }
It's quite roomy.
My vps conf:
[root@server1 ~]# cat /proc/sys/kernel/threads-max 256653 [root@server1 ~]# cat /proc/sys/kernel/pid_max 32768 [root@server1 ~]# ulimit unlimited
So I think is a pymongo problem.
in <module>
from udbswp.handler.api import spider_api
File "/mnt/apps/workspace/udb-spider/udbswp/handler/api.py", line 169, in <module>
spider_api = Api()
File "/mnt/apps/workspace/udb-spider/udbswp/handler/api.py", line 43, in __init__
self.db = connect_database(API_DB_URL)
File "/mnt/apps/workspace/udb-spider/pyspider/pyspider/webui/database/__init__.py", line 23, in connect_database
return SpiderSettingDB(url)
File "/mnt/apps/workspace/udb-spider/pyspider/pyspider/webui/database/mongodb.py", line 95, in __init__
self.conn = MongoClient(url, maxPoolSize=3000)
File "/usr/local/python27/lib/python2.7/site-packages/pymongo/mongo_client.py", line 398, in __init__
executor.open()
File "/usr/local/python27/lib/python2.7/site-packages/pymongo/periodic_executor.py", line 68, in open
thread.start()
File "/usr/local/python27/lib/python2.7/threading.py", line 745, in start
_start_new_thread(self.__bootstrap, ())
error: can't start new thread