[SERVER-8224] when lots of web server, there is too many connections Created: 18/Jan/13  Updated: 10/Dec/14  Resolved: 22/Mar/13

Status: Closed
Project: Core Server
Component/s: Networking, Sharding, Usability
Affects Version/s: 2.2.2
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: peanutgyz Assignee: Jeremy Mikola
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux sharded cluster


Issue Links:
Depends
depends on SERVER-1714 multiplex mongos -> mongod connecions... Closed
Duplicate
is duplicated by SERVER-9022 Enable mongos mode which releases Sha... Closed
Operating System: ALL
Participants:

 Description   

I hava 100 web server, each use nginx and php-fpm.
create 200 php-cgi process per web server.

when i use persistent connection to mongos, there is 2w connections to mongos.
because mongos keep last connection, use avail(shard_version.cpp, shardconnection.cpp),

there will be 2w connection from mongos to shard server.



 Comments   
Comment by Jeremy Mikola [ 22/Mar/13 ]

peanutgyz: Just to follow up, there have been some recent improvements to mongos for releasing connections back to the pool (SERVER-9022). While PHP and its app server environment is still going to require more connections than other languages which implement pooling (e.g. Java, Python), you may see an improvement with this feature.

Comment by Jeremy Mikola [ 23/Jan/13 ]

Within the scope of this issue, there is nothing to fix in mongos. As far as it's concerned, the connections are active and cannot be shared with other sessions. Once a PHP worker dies and releases its mongos connection, the mongod connections in mongos do return to the pool to be used again.

Manually closing connections in the PHP driver is unlikely yield any improvement and may very well reduce performance, since the driver will not be able to re-use connections across the worker's lifetime. Each web request would correspond to a new mongos connection in that case, and at peak usage you may still see N connections to mongos where N is the number of PHP workers. The solution is to either reduce the number of worker processes, or redesign your application such that each web request does not require its own database connection (e.g. with caching).

Comment by peanutgyz [ 23/Jan/13 ]

i have more than 20000 php-fpm worker, so if php-fpm worker use persistent connection to mongos there will be too many connections on mongod.

should i close connnection in my php-wroker each time ??

Is there a plan to change mongos code, so mongos can put connections back to pool ?

Comment by Jeremy Mikola [ 22/Jan/13 ]

While mongos does maintain a connection pool for mongod connections, those connections are utilized by a single mongos client connection until it is closed (i.e. by the PHP process/worker). Therefore, if you have 20,000 simultaneous web server connections, you can expect 20,000 connections for the mongod processes through all of the mongos processes combined. There will likely be a few more connections from mongos to each mongod, for sharding overhead and things like balancing.

Limiting the worker processes, or abstracting your connections to mongos (e.g. application-level caching) will be necessary to keep under the 20,000 limit.

Comment by peanutgyz [ 21/Jan/13 ]

"use avail(shard_version.cpp, shardconnection.cpp)"

in source code, shardconnection.cpp, each ShardConnection variable has one DBClientBase* avail, which used to keep last connection to the shard.

when client use persistent connection to mongos , mongos doesn't put last connection to mongod back to connection pool, just keep it.
So if there is 1000 persistent connection to mongos, then there is 1000 connections from mongos to mongod.

When i use 100 web server, per server has 200 php-fpm worker, running mongos instantce on each web server, then there will be 200 connection to each mongos ,

and 200 connection from each mongos to mongod. there will also will be 20000 connection on mongod.

My problem is if hava too many fpm worker, then i will come right to 20000 limit on mongod.

Comment by Jeremy Mikola [ 18/Jan/13 ]

Can you clarify what "2w" is, and what you meant by "use avail(shard_version.cpp, shardconnection.cpp)"?

Are you receiving connection limit errors in the mongos or mongod logs? If so, can you share relevant segments of those log files?

Based on your web server topology, you would come right to the limit of 20,000 connections if all FPM workers connect to the same mongos instance. Have you considered running a mongos instance on each application server? That, or alternatively with having several mongos instances behind a load balancer, are typically advised for production deployments.

Generated at Thu Feb 08 03:16:52 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.