CPU usage is around 50% at system level, memory is under pressure and I notice 95% memory being used in a few scenarios. Number of connections spike to 5000 or more with 75% doing reads and remaining attempting to write.
I am not sure why I am running into this issue under load.

Too many threads are already waiting for a connection
• Errors
• About
MongoDB.Driver.MongoConnectionExceptionToo many threads are already waiting for a connection
MongoDB.Driver.MongoConnectionException: Too many threads are already waiting for a connection
at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection(MongoDatabase database)
at MongoDB.Driver.MongoServer.AcquireConnection(MongoDatabase database, Boolean slaveOk)
at MongoDB.Driver.MongoServer.RequestStart(MongoDatabase initialDatabase)
at RelayHealth.Hl7.Vortex.ProcessResolvePatientRequest.receiveMessage_Old(Guid token, HttpContextBase context, String documentidentifier) in f:\Builds\HotFix\RelayHealth\dev\MiddleTier\HTTPModule\src\RelayHea

************************************

User=System

TimeUtc=2012-09-25T15:04:43.0948183-07:00

At this point I am not certain who can help with this. Under load we ran MongoDB issues multiple times so far. I think I can attempt to reproduce some of the issues occasionally if not always.

Thanks,
Praveen

From: Evans, Martin P
Sent: Tuesday, September 25, 2012 3:01 PM
To: Veldanda, Praveen; *DL RHC SCM
Cc: Gutierrez, Ernesto; Pascual, Gerard; Cordell, Ron; *DL RHC P&R; Sivakumar, Rajesh; Kumarasamy, Senthil; Hill, Gennifer C; Akemon, Scott; Snow, Matthew
Subject: RE: MongoDB connectivity issues in performance environemnt since morning

The ticket you logged refered to socket errors. This is a transport-level connectivity issue. Please assign to the appropriate team.

From: Veldanda, Praveen
Sent: Tuesday, September 25, 2012 3:00 PM
To: Evans, Martin P; *DL RHC SCM
Cc: Gutierrez, Ernesto; Pascual, Gerard; Cordell, Ron; *DL RHC P&R; Sivakumar, Rajesh; Kumarasamy, Senthil; Hill, Gennifer C; Akemon, Scott; Snow, Matthew
Subject: RE: MongoDB connectivity issues in performance environemnt since morning

Hi Martin,
Please look at Mantis# 0054320. Gerard cannot access any MongoDB node in Performance environment and replica set needs to be rebuilt as per him.
Matt/ Rajesh – I will keep you posted on updates regarding environment.
Thanks

From: Veldanda, Praveen
Sent: Tuesday, September 25, 2012 1:01 PM
To: Gutierrez, Ernesto; Pascual, Gerard
Cc: Cordell, Ron; *DL RHC P&R; Sivakumar, Rajesh; Kumarasamy, Senthil; Hill, Gennifer C
Subject: RE: MongoDB connectivity issues in performance environemnt since morning

I noticed RelayHealth Mongo service is not running on Mongo servers in perf. Environment and started them. These are set to start manually (should they be automatic??).

Below are the errors after I restarted this service:

Too many threads are already waiting for a connection
• Errors
• About
MongoDB.Driver.MongoConnectionExceptionToo many threads are already waiting for a connection
MongoDB.Driver.MongoConnectionException: Too many threads are already waiting for a connection
at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection(MongoDatabase database)
at MongoDB.Driver.MongoServer.AcquireConnection(MongoDatabase database, Boolean slaveOk)
at MongoDB.Driver.MongoServer.RequestStart(MongoDatabase initialDatabase)
at RelayHealth.Hl7.Vortex.ProcessResolvePatientRequest.receiveMessage_Old(Guid token, HttpContextBase context, String documentidentifier) in f:\Builds\HotFix\RelayHealth\dev\MiddleTier\HTTPModule\src\RelayHea

A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
• Errors
• About
System.Net.Sockets.SocketExceptionA connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
— End of inner exception stack trace —
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at MongoDB.Bson.IO.BsonBuffer.LoadFrom(Stream stream, Int32 count)
at MongoDB.Bson.IO.BsonBuffer.LoadFrom(Stream stream)
at MongoDB.Driver.Internal.MongoConnection.ReceiveMessage[TDocument](MongoServer server)
at MongoDB.Driver.Internal.MongoConnection.RunCommand(MongoServer server, String collectionName, QueryFlags queryFlags, CommandDocument command)
at MongoDB.Driver.Internal.MongoConnection.Authenticate(MongoServer server, String databaseName, MongoCredentials credentials)
at MongoDB.Driver.Internal.MongoConnection.CheckAuthentication(MongoServer server, MongoDatabase database)
at MongoDB.Driver.MongoServer.AcquireConnection(MongoDatabase database, Boolean slaveOk)
at MongoDB.Driver.MongoServer.RequestStart(MongoDatabase initialDatabase)
at RelayHealth.Hl7.Vortex.ProcessResolvePatientRequest.receiveMessage_Old(Guid token, HttpContextBase context, String documentidentifier) in f:\Builds\HotFix\RelayHealth\dev\MiddleTier\HTTPModule\src\RelayHealth.Hl7.Vortex\ProcessResolvePatientRequest.cs:line 317

***********************