[CSHARP-636] Mongo authentication issues under load - currently on MongoDB 2.2, Mongo drivers Created: 27/Nov/12  Updated: 16/Jan/14  Resolved: 04/Mar/13

Status: Closed
Project: C# Driver
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: praveen veldanda Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows 2008 OS R2 X64 edition ; 4 CPU, 16 GB mongo nodes in a non-production environment



 Description   

CPU usage is around 50% at system level, memory is under pressure and I notice 95% memory being used in a few scenarios. Number of connections spike to 5000 or more with 75% doing reads and remaining attempting to write.
I am not sure why I am running into this issue under load.

Too many threads are already waiting for a connection
• Errors
• About
MongoDB.Driver.MongoConnectionExceptionToo many threads are already waiting for a connection
MongoDB.Driver.MongoConnectionException: Too many threads are already waiting for a connection
at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection(MongoDatabase database)
at MongoDB.Driver.MongoServer.AcquireConnection(MongoDatabase database, Boolean slaveOk)
at MongoDB.Driver.MongoServer.RequestStart(MongoDatabase initialDatabase)
at RelayHealth.Hl7.Vortex.ProcessResolvePatientRequest.receiveMessage_Old(Guid token, HttpContextBase context, String documentidentifier) in f:\Builds\HotFix\RelayHealth\dev\MiddleTier\HTTPModule\src\RelayHea

************************************

User=System

TimeUtc=2012-09-25T15:04:43.0948183-07:00

At this point I am not certain who can help with this. Under load we ran MongoDB issues multiple times so far. I think I can attempt to reproduce some of the issues occasionally if not always.

Thanks,
Praveen

From: Evans, Martin P
Sent: Tuesday, September 25, 2012 3:01 PM
To: Veldanda, Praveen; *DL RHC SCM
Cc: Gutierrez, Ernesto; Pascual, Gerard; Cordell, Ron; *DL RHC P&R; Sivakumar, Rajesh; Kumarasamy, Senthil; Hill, Gennifer C; Akemon, Scott; Snow, Matthew
Subject: RE: MongoDB connectivity issues in performance environemnt since morning

The ticket you logged refered to socket errors. This is a transport-level connectivity issue. Please assign to the appropriate team.

From: Veldanda, Praveen
Sent: Tuesday, September 25, 2012 3:00 PM
To: Evans, Martin P; *DL RHC SCM
Cc: Gutierrez, Ernesto; Pascual, Gerard; Cordell, Ron; *DL RHC P&R; Sivakumar, Rajesh; Kumarasamy, Senthil; Hill, Gennifer C; Akemon, Scott; Snow, Matthew
Subject: RE: MongoDB connectivity issues in performance environemnt since morning

Hi Martin,
Please look at Mantis# 0054320. Gerard cannot access any MongoDB node in Performance environment and replica set needs to be rebuilt as per him.
Matt/ Rajesh – I will keep you posted on updates regarding environment.
Thanks

From: Veldanda, Praveen
Sent: Tuesday, September 25, 2012 1:01 PM
To: Gutierrez, Ernesto; Pascual, Gerard
Cc: Cordell, Ron; *DL RHC P&R; Sivakumar, Rajesh; Kumarasamy, Senthil; Hill, Gennifer C
Subject: RE: MongoDB connectivity issues in performance environemnt since morning

I noticed RelayHealth Mongo service is not running on Mongo servers in perf. Environment and started them. These are set to start manually (should they be automatic??).

Below are the errors after I restarted this service:

Too many threads are already waiting for a connection
• Errors
• About
MongoDB.Driver.MongoConnectionExceptionToo many threads are already waiting for a connection
MongoDB.Driver.MongoConnectionException: Too many threads are already waiting for a connection
at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection(MongoDatabase database)
at MongoDB.Driver.MongoServer.AcquireConnection(MongoDatabase database, Boolean slaveOk)
at MongoDB.Driver.MongoServer.RequestStart(MongoDatabase initialDatabase)
at RelayHealth.Hl7.Vortex.ProcessResolvePatientRequest.receiveMessage_Old(Guid token, HttpContextBase context, String documentidentifier) in f:\Builds\HotFix\RelayHealth\dev\MiddleTier\HTTPModule\src\RelayHea

A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
• Errors
• About
System.Net.Sockets.SocketExceptionA connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
— End of inner exception stack trace —
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at MongoDB.Bson.IO.BsonBuffer.LoadFrom(Stream stream, Int32 count)
at MongoDB.Bson.IO.BsonBuffer.LoadFrom(Stream stream)
at MongoDB.Driver.Internal.MongoConnection.ReceiveMessage[TDocument](MongoServer server)
at MongoDB.Driver.Internal.MongoConnection.RunCommand(MongoServer server, String collectionName, QueryFlags queryFlags, CommandDocument command)
at MongoDB.Driver.Internal.MongoConnection.Authenticate(MongoServer server, String databaseName, MongoCredentials credentials)
at MongoDB.Driver.Internal.MongoConnection.CheckAuthentication(MongoServer server, MongoDatabase database)
at MongoDB.Driver.MongoServer.AcquireConnection(MongoDatabase database, Boolean slaveOk)
at MongoDB.Driver.MongoServer.RequestStart(MongoDatabase initialDatabase)
at RelayHealth.Hl7.Vortex.ProcessResolvePatientRequest.receiveMessage_Old(Guid token, HttpContextBase context, String documentidentifier) in f:\Builds\HotFix\RelayHealth\dev\MiddleTier\HTTPModule\src\RelayHealth.Hl7.Vortex\ProcessResolvePatientRequest.cs:line 317

***********************



 Comments   
Comment by Peter Aberline [ 16/Jan/14 ]

Hi Praveen

What versions of the server and driver were you experiencing this issue with? And what versions are you now using which resolve the problem?

Thanks
Peter

Comment by praveen veldanda [ 04/Mar/13 ]

Hey Craig,
Can you remove this thread since I noticed there are emailids and names of my colleagues?
This problem is resolved since I moved to latest version of Mongo drivers and DB.
Thanks.

Comment by Craig Wilson [ 26/Dec/12 ]

Praveen -
Any further information on this? It's been a while and we'd like to close this ticket if it is no longer an issue.

Comment by Craig Wilson [ 29/Nov/12 ]

Praveen,
1) What version of the driver are you using?
2) You mention CPU at 50% and connections at 5000? Is this server side (mongod or mongos) or client-side (C# driver)?
3) Have you adjusted the connection pool size defaults? This defaults to 100 connections and allows 100 connections to be waiting for a connection. Therefore, the 201st connection would throw this exception if the defaults have not been adjusted?
4) I notice you are using RequestStart. Are you disposing of the RequestStart? If you don't dispose of this, then you will quickly run out of connections.

using(server.RequestStart())
{
    // do your thing...
}

Let me know...

Generated at Wed Feb 07 21:37:24 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.