MongoDB cluster is consist of 28 shards and 15 mongos.
There's about 4K/sec queries per shard and small DMLs and there's about 600~1000 connections(from mongos) per shard.
MongoDB connections graph is spiky only when user authentication is enabled Not with disabled authentication. And java client driver return errors(like below) or slow-responses when user authentication is enabled. (I attached connections and query/sec graphs).
[ERROR] [c.k.s.m.g.r.RequestExecuteCallable] execute (71): Too many threads are already waiting for a connection. Max number of threads (maxWaitQueueSize) of 400 has been exceeded.
com.mongodb.MongoWaitQueueFullException: Too many threads are already waiting for a connection. Max number of threads (maxWaitQueueSize) of 400 has been exceeded.
Not exactly 5 minutes, but connection spike(and client queue error) is happened about 5 minutes intervals.
I am not sure "ConnectionPool::kDefaultHostTimeout" cause this issue yet, But I think 5 minutes is too short to shutdown connection pool. it seems so aggressive. Is there any reason to choose 5 minutes to shutdown pool ?