[CSHARP-343] .Net Web App via C# driver reports connection problems Created: 18/Oct/11  Updated: 02/Apr/15  Resolved: 20/Feb/12

Status: Closed
Project: C# Driver
Component/s: None
Affects Version/s: 1.1
Fix Version/s: 1.3.1

Type: Bug Priority: Major - P3
Reporter: Dan Pinkard Assignee: Robert Stam
Resolution: Done Votes: 1
Labels: driver
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows Server 2008R2 Datacenter (EC2) running IIS7


Issue Links:
Duplicate
duplicates CSHARP-406 Deadlock and TimeoutException when ac... Closed

 Description   

Two symptoms, potentially linked:

  • Specific server in a farm of identical machines generates errors, apparently one per second It is logged Via elmah as:
    System.TimeoutException: Timeout waiting for a MongoConnection.
    at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection(MongoDatabase database)
    at MongoDB.Driver.MongoServerInstance.AcquireConnection(MongoDatabase database)
    at MongoDB.Driver.MongoServer.AcquireConnection(MongoDatabase database, Boolean slaveOk)
    at MongoDB.Driver.MongoCursorEnumerator`1.AcquireConnection()
    at MongoDB.Driver.MongoCursorEnumerator`1.GetFirst()
    at MongoDB.Driver.MongoCursorEnumerator`1.MoveNext()
    at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)
    at MongoDB.Driver.MongoCollection.FindOneAs[TDocument](IMongoQuery query)
    at MongoDB.Driver.MongoCollection`1.FindOne(IMongoQuery query)
    at MongoDB.Driver.MongoDatabase.RunCommandAs(Type commandResultType, IMongoCommand command)
    at MongoDB.Driver.MongoDatabase.RunCommandAs[TCommandResult](IMongoCommand command)
    at MongoDB.Driver.MongoDatabase.RunCommand(IMongoCommand command)
    at MongoDB.Driver.MongoCollection.Count(IMongoQuery query)

This is resolved by resetting the app-pool for that site on that host. No specific trigger is found on the servers themselves in either the mongodb.log files nor from other system monitoring tools.

  • Some time later (30 minutes)all servers in the pool (of 6) begin logging:
    System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
    at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
      • End of inner exception stack trace —
        at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
        at MongoDB.Bson.IO.BsonBuffer.LoadFrom(Stream stream, Int32 count)
        at MongoDB.Bson.IO.BsonBuffer.LoadFrom(Stream stream)
        at MongoDB.Driver.Internal.MongoConnection.ReceiveMessage[TDocument](BsonBinaryReaderSettings readerSettings, IBsonSerializationOptions serializationOptions)
        at MongoDB.Driver.MongoCursorEnumerator`1.GetReply(MongoConnection connection, MongoRequestMessage message)
        at MongoDB.Driver.MongoCursorEnumerator`1.GetFirst()
        at MongoDB.Driver.MongoCursorEnumerator`1.MoveNext()
        at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)
        at MongoDB.Driver.MongoCollection.FindOneAs[TDocument](IMongoQuery query)
        at MongoDB.Driver.MongoCollection`1.FindOne(IMongoQuery query)
        at MongoDB.Driver.MongoDatabase.RunCommandAs(Type commandResultType, IMongoCommand command)
        at MongoDB.Driver.MongoDatabase.RunCommandAs[TCommandResult](IMongoCommand command)
        at MongoDB.Driver.MongoDatabase.RunCommand(IMongoCommand command)
        at MongoDB.Driver.MongoCollection.Count(IMongoQuery query)

This is again resolved by an app-pool reset.

In both cases the evidence points more to the driver, however we would like to know how to go about directing ourselves in one direction or another. A problem local to one server doesn't feel like a server issues, however a problem shared among several could be.



 Comments   
Comment by Robert Stam [ 05/Mar/12 ]

CSHARP-406 fixes the most likely cause for seeing this exception. Note that under heavy loads you might still see this exception.

Comment by Robert Stam [ 20/Feb/12 ]

The "Timeout waiting for a MongoConnection" could happen after time due to a bug in the 1.2 version of the driver.

The IOException is caused by a timeout waiting for a response from the server. The default timeout is 30 seconds, so if you are getting timeouts it means some queries are taking longer than 30 seconds. I have also seen this exception happen when you configure the socketTimeout to a value that is too short.

I am closing this issue because I believe that it only affects version 1.2 and not the latest version of the driver.

Comment by Dan Pinkard [ 20/Oct/11 ]

Correction: We are on the 1.2 driver.

Comment by Dan Pinkard [ 18/Oct/11 ]

Will investigate moving up to 1.2. We are using the default connection pool size, however I am not sure how many threads are drawing from it.

It would be nice for the driver and/or application can bubble up the IP of the offending server(s), but I'm not sure who is dropping that at the moment.

Comment by Robert Stam [ 18/Oct/11 ]

v1.1 of the C# driver is one release back. Any chance you could upgrade to v1.2?

Both exceptions are symptoms of the server not responding quickly enough.

The first exception occurs because the driver times out waiting for a connection from the connection pool. How many threads do you have? When queries are slow then connections are also slow to be returned to the connection pool, which can lead to this exception. The default connection pool size is 100.

The second exception occurs when the driver does not receive a response from the server within the timeout period (default is 30 seconds).

I can't tell from the exceptions alone whether the driver has any role to play in the poor performance or not. v1.1 of the driver did have a negative feedback loop where once errors started occurring they got worse (see CSHARP-302). What you would see in the server log is lots of messages reporting that connections are being opened and closed.

Generated at Wed Feb 07 21:36:32 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.