[SERVER-28057] isMaster timeout when using multiple mongos connection pools Created: 19/Feb/17  Updated: 21/Feb/17  Resolved: 21/Feb/17

Status: Closed
Project: Core Server
Component/s: Networking, Sharding
Affects Version/s: 3.2.12
Fix Version/s: None

Type: Bug Priority: Minor - P4
Reporter: Antonis Giannopoulos Assignee: Unassigned
Resolution: Done Votes: 1
Labels: 3.2, ismaster, mongos, timeout
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

CentoOS 6
MongoDB version 3.2.12


Operating System: ALL
Steps To Reproduce:

Create a 3.2.12 sharded cluster with a single mongos (having 6 CPU), config servers on SCCC and 10 shards - 3 node replica set.
Using sysbench-mongodb create six test collections sbtest<1-6> 40.000.000 docs each and shard on {_id:hashed}
After balancing complete, execute four instances of sbench-mongodb against the mongos. I am running sbench-mongodb using defaults but NUM_WRITER_THREADS which i change it to 256.
After few minutes ASIO timeouts will start appearing on the mongos log.

Participants:

 Description   

I am testing 3.2.12 on a 10 nodes sharded cluster (using sysbench-mongodb) and I am getting a weird behavior. Whenever using mongos default settings I am receiving random ASIO timeouts for

{ isMaster: 1 }

command from different connection pools.

I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-2-0] Failed to connect to (node) - ExceededTimeLimit: Operation timed out
D ASIO     [NetworkInterfaceASIO-TaskExecutorPool-2-0] Failed to execute command: RemoteCommand 23628777 -- target:(node) db:admin cmd:{ isMaster: 1 } reason: ExceededTimeLimit: Operation timed out

When I set "taskExecutorPoolSize"=1, which I believe set a single connection pool, I am not getting the above errors.

My mongos has 6 CPUs so I assume it creates 6 connection pools with defaults. Using a smaller value like "taskExecutorPoolSize"=2 reduces the timeouts so it seems the more connection pools I use the more timeouts I get during the benchmark.

I am trying to understand what may cause the above behavior.

Thanks in advance,
Antonis



 Comments   
Comment by Kelsey Schubert [ 21/Feb/17 ]

Hi antogiann,

Thanks for your report. Please note that SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-users group.

Kind regards,
Thomas

Generated at Thu Feb 08 04:17:00 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.