[SERVER-8224] when lots of web server, there is too many connections Created: 18/Jan/13 Updated: 10/Dec/14 Resolved: 22/Mar/13 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Networking, Sharding, Usability |
| Affects Version/s: | 2.2.2 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | peanutgyz | Assignee: | Jeremy Mikola |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Linux sharded cluster |
||
| Issue Links: |
|
||||||||||||||||
| Operating System: | ALL | ||||||||||||||||
| Participants: | |||||||||||||||||
| Description |
|
I hava 100 web server, each use nginx and php-fpm. when i use persistent connection to mongos, there is 2w connections to mongos. there will be 2w connection from mongos to shard server. |
| Comments |
| Comment by Jeremy Mikola [ 22/Mar/13 ] |
|
peanutgyz: Just to follow up, there have been some recent improvements to mongos for releasing connections back to the pool ( |
| Comment by Jeremy Mikola [ 23/Jan/13 ] |
|
Within the scope of this issue, there is nothing to fix in mongos. As far as it's concerned, the connections are active and cannot be shared with other sessions. Once a PHP worker dies and releases its mongos connection, the mongod connections in mongos do return to the pool to be used again. Manually closing connections in the PHP driver is unlikely yield any improvement and may very well reduce performance, since the driver will not be able to re-use connections across the worker's lifetime. Each web request would correspond to a new mongos connection in that case, and at peak usage you may still see N connections to mongos where N is the number of PHP workers. The solution is to either reduce the number of worker processes, or redesign your application such that each web request does not require its own database connection (e.g. with caching). |
| Comment by peanutgyz [ 23/Jan/13 ] |
|
i have more than 20000 php-fpm worker, so if php-fpm worker use persistent connection to mongos there will be too many connections on mongod. should i close connnection in my php-wroker each time ?? Is there a plan to change mongos code, so mongos can put connections back to pool ? |
| Comment by Jeremy Mikola [ 22/Jan/13 ] |
|
While mongos does maintain a connection pool for mongod connections, those connections are utilized by a single mongos client connection until it is closed (i.e. by the PHP process/worker). Therefore, if you have 20,000 simultaneous web server connections, you can expect 20,000 connections for the mongod processes through all of the mongos processes combined. There will likely be a few more connections from mongos to each mongod, for sharding overhead and things like balancing. Limiting the worker processes, or abstracting your connections to mongos (e.g. application-level caching) will be necessary to keep under the 20,000 limit. |
| Comment by peanutgyz [ 21/Jan/13 ] |
|
"use avail(shard_version.cpp, shardconnection.cpp)" in source code, shardconnection.cpp, each ShardConnection variable has one DBClientBase* avail, which used to keep last connection to the shard. when client use persistent connection to mongos , mongos doesn't put last connection to mongod back to connection pool, just keep it. When i use 100 web server, per server has 200 php-fpm worker, running mongos instantce on each web server, then there will be 200 connection to each mongos , and 200 connection from each mongos to mongod. there will also will be 20000 connection on mongod. My problem is if hava too many fpm worker, then i will come right to 20000 limit on mongod. |
| Comment by Jeremy Mikola [ 18/Jan/13 ] |
|
Can you clarify what "2w" is, and what you meant by "use avail(shard_version.cpp, shardconnection.cpp)"? Are you receiving connection limit errors in the mongos or mongod logs? If so, can you share relevant segments of those log files? Based on your web server topology, you would come right to the limit of 20,000 connections if all FPM workers connect to the same mongos instance. Have you considered running a mongos instance on each application server? That, or alternatively with having several mongos instances behind a load balancer, are typically advised for production deployments. |