[SERVER-41557] Booming connections on primary mongod in sharded cluster Created: 05/Jun/19 Updated: 27/Oct/23 Resolved: 10/Jun/19 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Question | Priority: | Major - P3 |
| Reporter: | Zhexuan Chen | Assignee: | Dmitry Agranat |
| Resolution: | Community Answered | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Participants: |
| Description |
|
Our structure: we have 3-sharded cluster, has ~8 mongoS and 20 mongod. We experience the booming connections on primary mongod in one of the shards. The mongod operation logs show thousands of these logs.
The total connection number jump from ~800 to ~5000 in two minutes. During this 2 minute, no other commands get executed, and after these booming connections, bunches of WRITE operation finished, with database intent exclusive lock timeAcquiringMicros: { w: 135344866 }. Looks like the problem is all WRITE is waiting for database intent exclusive lock, and have contention with some other resources. My questions are: what is that driver name: "MongoDB Internal Client" meaning? It should not be the mongos -> mongod connection. They should be something like "NetworkInterfaceASIO-TaskExecutorPool-3". Also, it should not be replication from secondary to primary, because I also see some other connections that mean these: "NetworkInterfaceASIO-Replication", from mongodb doc.
Thank you very much for the help and really appreciate on any helps!
|
| Comments |
| Comment by Dmitry Agranat [ 06/Jun/19 ] |
|
Hi zchen12345, MongoDB Internal Client means this is one of the Cluster members, I suggest mapping all the involved IPs to figure out the source. Please note that SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-user group. Thank you, |