[CSHARP-1303] Socket Connection Error in 2.0 Driver Created: 09/Jun/15  Updated: 15/Nov/21  Resolved: 04/Apr/16

Status: Closed
Project: C# Driver
Component/s: Connectivity, Error Handling
Affects Version/s: 2.0
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Bret Ferrier Assignee: Unassigned
Resolution: Done Votes: 2
Labels: question
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Azure website connecting to a Linux replica set on Azure


Attachments: PNG File screenshot-1.png     HTML File text.html    
Issue Links:
Related
related to CSHARP-1994 Enable and configure TCP Keepalive by... Closed

 Description   

Since going through and updating all of my code to use the mongodb 2.0 driver when I look in my logs I see hundreds of error messages just like the one below where the driver is getting a socket error when connecting to mongo. When I look at the logs on the mongo server I do not see any error messages. The site is still up and this error doesn't happen on every request, nor does it happen on one operation that should take longer. It is random where it occurs within the application.

Here is the error:

MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 138.91.138.169:27017
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.<ConnectAsync>d__7.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.<CreateStreamAsync>d__0.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenAsyncHelper>d__1.MoveNext()
— End of inner exception stack trace —
at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenAsyncHelper>d__1.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.Core.Servers.ClusterableServer.<GetChannelAsync>d__0.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at MongoDB.Driver.Core.Operations.FindOperation`1.<ExecuteAsync>d__2.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.OperationExecutor.<ExecuteReadOperationAsync>d__0`1.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.MongoCollectionImpl`1.<ExecuteReadOperation>d__35`1.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MongoDB.Driver.IAsyncCursorSourceExtensions.<ToListAsync>d__14`1.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at CitySpark.EventCache.FlatEventRepo.<GetEventsPagedAsync>d__26.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()



 Comments   
Comment by John Murphy [ 16/Sep/19 ]

If you are unable to enable TCP keep alives at the Operating System or Socket level, you can set the MaxConnectionIdleTime to instruct the MongoDB C#/.NET driver to not use pooled connections that may have been severed by the Azure Network Load Balancer.

This approach however will not proactively detect connections that have been reset unexpectedly, and it will not prevent connections with long running operations from being severed by Azure.

Therefore we suggest that you utilise the TCP keep alive method when using the MongoDB C#/.NET driver within Azure environments to provide the highest level of connection resilience.

Comment by Sebastian Inones [ 12/Sep/19 ]

I'm facing the same issue using App Service hosted on Azure connected to a MongoDB Server outside Azure on a different place.
I have read what @John Murphy said. My question is (I'm sort of a newbie in this kind of connection issues/scenarios):
Wouldn't be enough to set:

mongoClientSettings.MaxConnectionIdleTime = TimeSpan.FromMinutes(3);

if as he mentioned "environments such as Azure sever idle TCP connections after 4 minutes." ?

Comment by John Murphy [ 20/Jun/19 ]

Note that our production notes state that environments such as Azure sever idle TCP connections after 4 minutes. Therefore in these environments the keep alive interval needs to be configured to be 120 seconds.

This comment on CSHARP-2543 shows how to set the interval to 120 seconds.

Comment by Vadim Rybak [ 30/Aug/17 ]

Even this workaround doesn't seem to be working for us.
We use structuremap to make sure that our `IMongoClient` is a singleton. Here is how we have it setup:

For<IMongoClient>().Use("mongoClientCreator", ctx => {
	var conn = ctx.GetInstance<IOptions<ConnectionStrings>>().Value.Mongo;
 
	var settings = MongoClientSettings.FromUrl(MongoUrl.Create(conn));
	settings.ConnectTimeout = TimeSpan.FromSeconds(60);
	settings.SocketTimeout = TimeSpan.FromMinutes(5);
	settings.MaxConnectionIdleTime = TimeSpan.FromSeconds(30);
	void SocketConfigurator(Socket s) => s.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.KeepAlive, true);
 
	settings.ClusterConfigurator = builder => builder
		.ConfigureTcp(tcp => tcp.With(socketConfigurator: (Action<Socket>)SocketConfigurator));
 
	return new MongoClient(settings);
})
.Singleton();

We also confirmed that our keep alive on the box hosting our code is set correctly

We are still getting the same issues that Jeremy was.
The interesting thing is when we connected to a single mongo machine in Azure this was not an issue. It only when we started connecting to a replica set, that this became a problem for us.

Are there any other workarounds that we can do?

Comment by Jeffrey Yemin [ 29/Aug/17 ]

jstafford@digitalairstrike.com thanks for posting your workaround, and also please watch this related ticket to enable TCP Keep-Alive by default: CSHARP-1994.

Comment by Jeremy Stafford [ 29/Aug/17 ]

For what it's worth, it appears that the driver doesn't enable socket keep alive by default (strange considering this is a database connection...). The following solved it for us:

  • Set the duration and interval for socket keep alives in the VM (registry setting)
  • Enable socket keep alive in the C# driver like so:

_settings = new MongoClientSettings
{
    // our default settings
    ConnectTimeout = TimeSpan.FromSeconds(60),
    SocketTimeout = TimeSpan.FromSeconds(60),
    MaxConnectionIdleTime = TimeSpan.FromSeconds(60)
};
 
// enable keep-alives
void SocketConfigurator(Socket s) => s.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.KeepAlive, true);
 
_settings.ClusterConfigurator = builder =>
    builder.ConfigureTcp(tcp => tcp.With(socketConfigurator: (Action<Socket>)SocketConfigurator));

I'm surprised the MongoDB team isn't taking this more seriously. This stuff should be perfect for common use out of the box. To be fair though, apparently MSSQL had the same issue in Azure a while back and solved it in a similar fashion in the ADO driver.

Comment by Jaume Bresco Arroyo [ 29/Aug/17 ]

We are experiencing the same issue, why is closed?

Thanks

Comment by Kamil Mackow [X] [ 28/Jul/17 ]

We are still experiencing this issue in Azure and have tried all suggestions thus far to no avail. Why is this closed?

Comment by Wayne He [ 06/Jul/17 ]

I wonder why this bug is marked as Closed.
I am getting the same error frequently with new 2.x driver. I have updated to the latest C# driver, v2.4.4. My MongoDB server, 3.4.4, is a small shard with replset of 3, running on Linux hosts in local corporate LAN.

Comment by Jeremy Stafford [ 10/Feb/17 ]

So by my reading, Azure's default timeout is 4 minutes. I set maxIdleTime to 60000 and 10000 and still getting the same problem.

Comment by Craig Wilson [ 03/Feb/16 ]

Sinan,

Azure has a low setting for idle connections and kills them when they've hit that limit. You need to specify either maxIdleTime on the connection string or MaxConnectionIdleTime in MongoClientSettings to something around 3 minutes. That way, the driver won't use a connection that has been idle for too long and may have been killed off by azure already.

Craig

Comment by Sinan Bir [ 03/Feb/16 ]

I have Linux replica set on Azure and my front-end application deployed as a cloud service in Azure. When I send request after a while ( about 10 minutes later after first request ) I always get socket exception error.

[SocketException (0x274c): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond]
System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult) +8451915
System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult) +48

[IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.]
System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult) +491
System.Threading.Tasks.FromAsyncTrimPromise`1.Complete(TInstance thisRef, Func`3 endMethod, IAsyncResult asyncResult, Boolean requiresSynchronization) +33
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13877064
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
MongoDB.Driver.Core.Misc.<ReadBytesAsync>d__2.MoveNext() +541
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13877064
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
MongoDB.Driver.Core.Connections.<ReceiveBufferAsync>d__50.MoveNext() +390

[MongoConnectionException: An exception occurred while receiving a message from the server.]
MongoDB.Driver.Core.Connections.<ReceiveBufferAsync>d__50.MoveNext() +1079
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13877064
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
MongoDB.Driver.Core.Connections.<ReceiveBufferAsync>d__51.MoveNext() +1351
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13877064
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
MongoDB.Driver.Core.Connections.<ReceiveMessageAsync>d__53.MoveNext() +765
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13877064
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
MongoDB.Driver.Core.WireProtocol.<ExecuteAsync>d__11.MoveNext() +887
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13877064
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
MongoDB.Driver.Core.Servers.<ExecuteProtocolAsync>d__26`1.MoveNext() +542
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13877064
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
MongoDB.Driver.Core.Operations.<ExecuteProtocolAsync>d__109.MoveNext() +739
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13877064
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
MongoDB.Driver.Core.Operations.<ExecuteAsync>d__107.MoveNext() +1374
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13877064
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
MongoDB.Driver.Core.Operations.<ExecuteAsync>d__107.MoveNext() +1332
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13877064
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
MongoDB.Driver.<ExecuteReadOperationAsync>d__1`1.MoveNext() +377
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13877064
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
MongoDB.Driver.<ExecuteReadOperationAsync>d__59`1.MoveNext() +472
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13877064
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
Repository.IRepository.<GetById>d__31.MoveNext() +423
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13877064
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
Service.Service.<FindClient>d__11.MoveNext() +214
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13877064
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
MMT.WebApi.AuthProviders.<ValidateClientAuthentication>d__3.MoveNext() +386
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13877064
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
Microsoft.Owin.Security.OAuth.<InvokeTokenEndpointAsync>d__22.MoveNext() +866
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13877064
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
Microsoft.Owin.Security.OAuth.<InvokeAsync>d__0.MoveNext() +1211
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13877064
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
Microsoft.Owin.Security.Infrastructure.<Invoke>d__0.MoveNext() +540
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13877064
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.<RunApp>d__5.MoveNext() +202
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13877064
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.<DoFinalWork>d__2.MoveNext() +193
Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.StageAsyncResult.End(IAsyncResult ar) +96
System.Web.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +509
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +146

Comment by Roberto Pérez [ 02/Feb/16 ]

Same issue here.

Error Message:
A threw exception: MongoDB.Driver.MongoConnectionException: An exception occurred while receiving a message from the server. ---> System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

Stack Trace:
System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
System.IO.Stream.<>c_DisplayClass0.<BeginEndReadAsync>b_9(Stream stream, IAsyncResult asyncResult)
System.Threading.Tasks.TaskFactory`1.FromAsyncTrimPromise`1.Complete(TInstance thisRef, Func`3 endMethod, IAsyncResult asyncResult, Boolean requiresSynchronization)
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
MongoDB.Driver.Core.Misc.StreamExtensionMethods.<ReadBytesAsync>d__0.MoveNext()
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
MongoDB.Driver.Core.Connections.BinaryConnection.<ReceiveBufferAsync>d__45.MoveNext()

Thank you!

Comment by Bret Ferrier [ 24/Aug/15 ]

Ahmet, I ended up putting the MaxConnectionIdleTime around 45 seconds and then had to create a set of Extension Methods off of the IFindFluent Interface that included retry logic. In my testing trying the same query right after the exception was received just about always worked. Really seems that the retry logic or error handling should be baked in but it is not and doesn’t seem that anyone is looking at it.

From: ahmet taha sakar (JIRA)
Sent: Friday, August 21, 2015 6:39 AM
To: Bret Ferrier
Subject: [MongoDB-JIRA] (CSHARP-1303) Socket Connection Error in 2.0 Driver

[ https://jira.mongodb.org/browse/CSHARP-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1011590#comment-1011590 ]

ahmet taha sakar commented on CSHARP-1303:
------------------------------------------

I am also having the same issue. I'm on azure. I have default MaxConnectionIdleTime set to 1 minutes from the mongo c# driver.

----------------------
This message was sent from MongoDB's issue tracking system. To respond to this ticket, please login to https://jira.mongodb.org using your JIRA or MMS credentials.

Comment by Ahmet Taha sakar [ 21/Aug/15 ]

I am also having the same issue. I'm on azure. I have default MaxConnectionIdleTime set to 1 minutes from the mongo c# driver.

Comment by Bret Ferrier [ 09/Jun/15 ]

Below is the connection string

<add name="mongoEvents" connectionString="mongodb://remote:7CrazyIdea@xxmongolinux1.cloudapp.net:27017,xxmongolinux2.cloudapp.net:27017/Events?connectTimeoutMS=30000&socketTimeoutMS=30000&waitQueueTimeoutMS=30000&maxIdleTimeMS=45000" />

As you can see I am already setting the maxIdleTimeMS, On another site that we have I found that setting the maxIdleTime to 60000 didn't actually help but 45000 did solve the idle connection issue on Azure but this seems different.

-Bret

Comment by Craig Wilson [ 09/Jun/15 ]

What does your connection string look like? Or your in code setup?

Azure has something that tends to kill off idle connections. Please try setting the maxIdleTimeMS connection string option to 60000 (1 minute). You can also set this in code using the MaxConnectionIdleTime property on MongoClientSettings. Hopefully, this will clear it up.

Craig

Comment by Bret Ferrier [ 09/Jun/15 ]

Pretty much all of the exceptions I am seeing are the Socket exceptions.  Also this is running as an Azure website. 

Is there anything I can do with the Mongo driver to handle these exceptions?

Comment by Craig Wilson [ 09/Jun/15 ]

Hi Bret,

Sorry you are having issues. This exception is coming straight out of the .NET framework's Socket class. I assume one your mongod servers is sitting at 138.91.138.169:27017. Generally when this exception bubbles to user code, it means that the network is a little flaky. Otherwise, you would also probably be seeing a number of TimeoutExceptions in the logs as well. Is this the only exception you are seeing or are you also seeing TimeoutExceptions?

Also, are you using Azure or AWS?

Craig

Generated at Wed Feb 07 21:39:13 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.