[SERVER-41557] Booming connections on primary mongod in sharded cluster Created: 05/Jun/19  Updated: 27/Oct/23  Resolved: 10/Jun/19

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Zhexuan Chen Assignee: Dmitry Agranat
Resolution: Community Answered Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Participants:

 Description   

Our structure:

we have 3-sharded cluster, has ~8 mongoS and 20 mongod. We experience the booming connections on primary mongod in one of the shards. The mongod operation logs show thousands of these logs.

2019-05-22T20:00:38.826-0500 I NETWORK [conn7381256] received client metadata from 10.112.1.181:51422 conn7381256: { driver: { name: "MongoDB Internal Client", version: "3.4.14" }, os: { type: "Linux", name: "CentOS release 6.10 (Final)", architecture: "x86_64", version: "Kernel 2.6.32-754.9.1.el6.x86_64" } }

The total connection number jump from ~800 to ~5000 in two minutes. During this 2 minute, no other commands get executed, and after these booming connections, bunches of WRITE operation finished, with database intent exclusive lock timeAcquiringMicros: { w: 135344866 }.

Looks like the problem is all WRITE is waiting for database intent exclusive lock, and have contention with some other resources. My questions are: what is that driver name: "MongoDB Internal Client" meaning? 

It should not be the mongos -> mongod connection. They should be something like "NetworkInterfaceASIO-TaskExecutorPool-3".

Also, it should not be replication from secondary to primary, because I also see some other connections that mean these: "NetworkInterfaceASIO-Replication", from mongodb doc.

2019-05-22T19:48:46.856-0500 I NETWORK [conn7380126] received client metadata from 10.40.0.177:37562 conn7380126: { driver: { name: "NetworkInterfaceASIO-Replication", version: "3.4.14" }, os: { type: "Linux", name: "CentOS release 6.9 (Final)", architecture: "x86_64", versi
on: "Kernel 2.6.32-696.23.1.el6.x86_64" } }

Thank you very much for the help and really appreciate on any helps!

 



 Comments   
Comment by Dmitry Agranat [ 06/Jun/19 ]

Hi zchen12345,

MongoDB Internal Client means this is one of the Cluster members, I suggest mapping all the involved IPs to figure out the source.

Please note that SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-user group.

Thank you,
Dima

Generated at Thu Feb 08 04:58:04 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.