Priority: Major - P3
Affects Version/s: 3.4.4
Fix Version/s: None
After examining connections between mongos and mongod server, I've found these connections are created when they need even I added setParameter.ShardingTaskExecutorPoolMinSize on mongos.conf.
Mongos does not make connections until mongos receives some requests from client.
A lot of web service, we deploy new code with rolling restart of web servers (sometimes with restarting mongos also). Once we have restarted web server, user requests are come in with same amount of requests before restart. But mongos does't have any connections to mongod server, so mongos have to prepare lots of connections at once.
This leads lots of errors (like com.mongodb.MongoSocketReadTimeoutException and com.mongodb.MongoWaitQueueFullException) on client side. There's no easy way to warming-up connections between mongos and mongod because mongos maintains multi-layer connection pools (TaskExecutorPool and SpecificPool).
So, I think it would be better that mongos prepare connections as many as "setParameter.ShardingTaskExecutorPoolMinSize" when they start. And then we can do rolling-restart app/web server and mongos smoothly.
I've attached mongod connection graph(3 shard cluster, each line is for primary member of each shard) and client timeout (over 3 seconds) error graph.
You can see the timeout error and mongod connection spikes are moving together from two graphs.
And I hope this could be fixed ASAP, because critical in our service. If this is not fixed, we need to deploy new service code at night. I think other users also experience the same case if they are using rolling deployment also.