[SERVER-38220] When a mongod is dropped, mongos keeps retrying to connect to mongod even if the client has killed the connection. Created: 21/Nov/18 Updated: 27/Oct/23 Resolved: 26/Nov/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Networking, Sharding |
| Affects Version/s: | 3.4.18 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Feng Liyuan [X] | Assignee: | Danny Hatcher (Inactive) |
| Resolution: | Works as Designed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Operating System: | ALL | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Steps To Reproduce: | 1. my environment is
my client is a go program that do a single query
2. drop all connection from mongos to mongod
3. run the client, the query keeps about 1.5 second and meets SocketTimeout, then the client killed the connection.
4. But mongos keeps the connection 13 seconds, trys to connect to mongod 4 times.
5. When retrying, mongos keeps a CLOSE_WAIT socket with the client.
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Participants: |
| Description |
|
When a mongod is dropped, mongos keeps retrying to connect to mongod even if the client has killed the connection. The mongos rapidly inrcrease its open files if mongos do not stop retrying to connnect to mongod if it has got a FIN from the client. |
| Comments |
| Comment by Feng Liyuan [X] [ 27/Nov/18 ] | ||
|
Hi Daneil, In our production environment, the client keeps to query mongos in 2000 concurrency. If one mongd occurs something wrong, all mongos would keep thousands of CLOSE_WAIT connections to reach to fd limit. Is it more graceful if the second connection should stop retrying if the first connection has gone.
Besides, is there an option to control the connection timeout from mongos to mongod? The mongos' log said a creation of one connection took 98 seconds.
And is there an option to control the kMaxNumFailedHostRetryAttempts?
| ||
| Comment by Danny Hatcher (Inactive) [ 26/Nov/18 ] | ||
|
Hello Feng, There are two distinct connections in use during this scenario. The first connection is between the driver and the mongos, the second connection is between the mongos and the mongod. Apart from a few options such as maxTimeMS, the driver has little control over the connection between the mongos and the mongod. If the mongos can no longer contact the mongod, it will retry connections regardless of the driver's status. If you are looking at configuring the connection options between the mongos and the mongod, I recommend you read the documentation for the various ShardingTaskExecutorPool settings. Given the above, I will close this ticket as working as designed. Thank you, Danny |