[SERVER-6056] memory leak. query time out when sharding. Created: 11/Jun/12 Updated: 06/Dec/22 Resolved: 21/Mar/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 2.0.5, 2.0.6, 2.2.3 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | peanutgyz | Assignee: | [DO NOT USE] Backlog - Sharding Team |
| Resolution: | Done | Votes: | 1 |
| Labels: | connection, mongos, sharding | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
linux , 3 shard, each 3 server replicaset. |
||
| Assigned Teams: |
Sharding
|
| Operating System: | ALL |
| Participants: |
| Description |
|
mognos: because i use short connection in web client , each time client close connection , mongos destroy shardconnection and cause _map increate. sometimes mongos create new connection to mongod, with same pointer position with an old connection object, cause the new connection can get an positive sequence from _map , of 0!. it will cause checkShardVersion run >2 times, and cause web client wait 3 -5 seconds. i modify shard_version.cpp resetShardVersion function , when conn type is SET, connectionShardStatus.reset( getVersionable(conn) ); , fix this problem. |
| Comments |
| Comment by Gregory McKeon (Inactive) [ 21/Mar/18 ] | ||||||||||||||
|
Sharded queries no longer use this codepath. Please reopen if this is still an issue. | ||||||||||||||
| Comment by Greg Studer [ 13/Aug/13 ] | ||||||||||||||
|
There's been progress, but we're currently working on refactoring that entire codepath, which should make this better long-term. The workaround for now is to use connection pooling in the driver to avoid creating lots of new connections to mongos. | ||||||||||||||
| Comment by Jin-wook Jeong [ 09/Aug/13 ] | ||||||||||||||
|
any progress on this issue? After applying the peanutgyz's patch to version_manager.cpp of v2.4.5, the memory leak has gone away.
| ||||||||||||||
| Comment by Spencer Brody (Inactive) [ 22/Feb/13 ] | ||||||||||||||
|
Hi peanutgyz. I have taken a closer look at the code and agree with you now that there is a potential memory leak here. I am currently discussing the best way to fix it with my colleagues. Thank you for your help bringing this to our attention! | ||||||||||||||
| Comment by peanutgyz [ 23/Jan/13 ] | ||||||||||||||
|
can someone give me response ? | ||||||||||||||
| Comment by peanutgyz [ 16/Jan/13 ] | ||||||||||||||
|
1. I found there is 2 connection pool, one is defined at client/connpool.h DBConnectionPool pool; another is defined at s/shardconnection.cpp DBConnectionPool shardConnectionPool; 2. i change the code to PoolStats() : Command( "connPoolStats" ) {} command, which just call shardConnectionPool.appendInfo( result );, to check shardConnectionPool info, "hosts" : { , }, map size is 802 now . after i close all client connection , output change to : "hosts" : { , }, but map size is still 802. map size never scale down , 3. i think, mongos keep connection pool to all shard, but each pool has _maxPerHost = 50 limit . when a lot of client connection to mongos at same time , each client connection pop connection in shardConnection destruct func, will call versionManager.resetShardVersionCB( _conn ). _map.erase(conn) always return 0, never delete item from map. because when shardConnection destruct, it use ptr to shardConnection to reset map, | ||||||||||||||
| Comment by Spencer Brody (Inactive) [ 15/Jan/13 ] | ||||||||||||||
|
Mongos keeps a connection pool to all the shards. This pool does not scale down as the number of incoming client connections goes down - all connections opened remain opened forever unless there is a problem on that socket. That makes me think this isn't a bug. You can see information about the connections in the connection pool by running the "connPoolStats" command. So long as the size of _map remains relative to the number of connections in the connection pool, I don't think there is a memory leak. Can you check the size of the connection pool using connPoolStats and see if the size of _map scales with the size of the connection pool? | ||||||||||||||
| Comment by peanutgyz [ 15/Jan/13 ] | ||||||||||||||
|
3 Tue Jan 15 17:24:18 [conn2013] _map size is 3769 conn is 0x7f819c24be60, sequence is 0 | ||||||||||||||
| Comment by peanutgyz [ 15/Jan/13 ] | ||||||||||||||
|
S getSequence( DBClientBase * conn , const string& ns ) { I change the shard_version.cpp , add log to watch _map size. Then, i create 500 concurrent connection to mongos , query data, and _map incr. | ||||||||||||||
| Comment by Spencer Brody (Inactive) [ 11/Sep/12 ] | ||||||||||||||
|
I'm resolving this ticket due to lack of activity. If you have a test that demonstrates this problem, feel free to re-open. | ||||||||||||||
| Comment by Spencer Brody (Inactive) [ 20/Jun/12 ] | ||||||||||||||
|
Do you have a test that reproduces the problem? |