[SERVER-20709] mongoD runs out of connections very quickly, then becomes wedged Created: 30/Sep/15  Updated: 02/Oct/15  Resolved: 02/Oct/15

Status: Closed
Project: Core Server
Component/s: Admin
Affects Version/s: 3.0.5, 3.0.6
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Dai Shi Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Participants:

 Description   

We have a large sharded cluster where occasionally a secondary will become very slow and requires a restart to fix. All our reads go to secondaries. When inspecting the log of the secondary, I see that over the course of a couple seconds the number of connections goes up by many thousands. Then it will be filled with slow queries (taking over 100ms), which looks like every query hitting the replica. In normal operation these queries only take a few ms. After that I see lots of this sprinkled between the slow queries:

[conn233831] killcursors  keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { r: 2 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 52009 } }, Database: { acquireCount: { r: 1 } }, Collection: {
acquireCount: { r: 1 } } } 83ms

Putting the replica into maintenance mode (sometimes for many hours) and then putting it back into service does not fix the issue. After putting back into service, the node still continues to serve very slowly. Restarting the mongoD process however does fix the problem. We have experienced this in version 3.0.5 and 3.0.6, with both mmapv1 and wiredTiger storage engines.



 Comments   
Comment by Ramon Fernandez Marina [ 02/Oct/15 ]

Thanks for your report dai@foursquare.com. Please note that the SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag, where your question will reach a larger audience. A question like this involving more discussion would be best posted on the mongodb-user group. See also our Technical Support page for additional support resources.

Regards,
Ramón.

Generated at Thu Feb 08 03:55:02 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.