[CSHARP-2543] An exception occurred while receiving a message from the server Created: 13/Mar/19  Updated: 02/Apr/20  Resolved: 02/Apr/20

Status: Closed
Project: C# Driver
Component/s: Connectivity, Read Operations
Affects Version/s: 2.7.0
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Mitereiter Balazs Zoltan Assignee: Wan Bachtiar
Resolution: Done Votes: 0
Labels: question
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

windows, console application, .net framework 4.7.2


Case:

 Description   

We are running simultaneously approx. 15-16 instances of a tool (we wrote), which sometimes are throwing the following error messages by a read operation:

2019-03-13 00:02:59 An exception occurred while receiving a message from the server.
 MongoDB.Driver.Core
 ReceiveBuffer
 at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBuffer()
 at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBuffer(Int32 responseTo, CancellationToken cancellationToken)
 at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveMessage(Int32 responseTo, IMessageEncoderSelector encoderSelector, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)
 at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquiredConnection.ReceiveMessage(Int32 responseTo, IMessageEncoderSelector encoderSelector, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)
 at MongoDB.Driver.Core.WireProtocol.CommandUsingCommandMessageWireProtocol`1.Execute(IConnection connection, CancellationToken cancellationToken)
 at MongoDB.Driver.Core.WireProtocol.CommandWireProtocol`1.Execute(IConnection connection, CancellationToken cancellationToken)
 at MongoDB.Driver.Core.Servers.Server.ServerChannel.ExecuteProtocol[TResult](IWireProtocol`1 protocol, CancellationToken cancellationToken)
 at MongoDB.Driver.Core.Servers.Server.ServerChannel.Command[TResult](ICoreSession session, ReadPreference readPreference, DatabaseNamespace databaseNamespace, BsonDocument command, IEnumerable`1 commandPayloads, IElementNameValidator commandValidator, BsonDocument additionalOptions, Action`1 postWriteAction, CommandResponseHandling responseHandling, IBsonSerializer`1 resultSerializer, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)
 at MongoDB.Driver.Core.Operations.AsyncCursor`1.ExecuteGetMoreCommand(IChannelHandle channel, CancellationToken cancellationToken)
 at MongoDB.Driver.Core.Operations.AsyncCursor`1.GetNextBatch(CancellationToken cancellationToken)
 at MongoDB.Driver.Core.Operations.AsyncCursor`1.MoveNext(CancellationToken cancellationToken)
 at MongoDB.Driver.Core.Operations.AsyncCursorEnumerator`1.MoveNext()

In the collection we have 74160665 docs (avg. doc size 1.7KB). The query uses a multikey compound index with 5 fields (string, bool, bool, bool, array) and we are using projection to acquire the _id and a string type field. Our cluster consists of 3 data-bearing nodes and we use nearest readPreference. Some of the other tool instances are making writes on this collection parallel. The method which makes the read operation is executed in a synchronous fashion, and we tried to read the results into a cursor.
What could cause this behavior? If more info is needed I will try to provide it.

Sorry for the previous posting (2542), I accidentally pressed the enter key.



 Comments   
Comment by John Murphy [ 20/Jun/19 ]

MongoDB driver version 2.7.2 saw CSHARP-1994 introduced which turns on TCP keep alives by default. These keep alives have an interval of 300 seconds, as defined by the MongoDB drivers specification.

This interval is acceptable for most networks, but as joshua.maag noted previously some cloud environments like Azure need a more aggressive interval set. In these network environments we advise that the interval be reduced to 120 seconds, to work around the cloud provider load balancers that sever connections it believes are idle.

As per our production notes you can either configure the system TCP keepalive setting using the HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\KeepAliveTime registry key, or at runtime by implementing the following Socket configurator within your application code:

public class Program
{
	internal struct KeepAliveValues
	{
		public uint OnOff { get; set; }
		public uint KeepAliveTime { get; set; }
		public uint KeepAliveInterval { get; set; }
 
		public byte[] ToBytes()
		{
			var bytes = new byte[12];
			Array.Copy(BitConverter.GetBytes(OnOff), 0, bytes, 0, 4);
			Array.Copy(BitConverter.GetBytes(KeepAliveTime), 0, bytes, 4, 4);
			Array.Copy(BitConverter.GetBytes(KeepAliveInterval), 0, bytes, 8, 4);
			return bytes;
		}
	}
 
	internal static void SocketConfigurator(Socket s)
	{
		var keepAliveValues = new KeepAliveValues()
		{
			OnOff = 1,
			KeepAliveTime = 120 * 1000,   // 120 seconds in milliseconds
			KeepAliveInterval = 10 * 1000 // 10 seconds in milliseconds
		};
 
		s.IOControl(IOControlCode.KeepAliveValues, keepAliveValues.ToBytes(), null);
	}
	
	public void Main(string[] args)
	{
		var url = new MongoUrl("mongodb://localhost");
		var clientSettings = MongoClientSettings.FromUrl(url);
		clientSettings.ClusterConfigurator = cb => cb.ConfigureTcp(tcp => tcp.With(socketConfigurator: (Action<Socket>)SocketConfigurator));
		var client = new MongoClient(clientSettings);
	}
}

 

 

Comment by Joshua Maag [ 16/Jun/19 ]

Can we get the TCP Keepalive for both the client and server for these scenarios?  Doing some investigation, it appears this issue can be caused if the TCP Keepalive does not follow our production guidelines

We've also seen some scenarios in Azure worth mentioning here:

The TCP idle timeout on the Azure load balancer is 240 seconds by default, which can cause it to silently drop connections if the TCP keepalive on your Azure systems is greater than this value. You should set tcp_keepalive_time to 120 to ameliorate this problem. 

Additionally, it's also worth looking at CSHARP-1303 if your running in Azure for additional options in configuring the driver to handle those timeouts.  

Comment by Nazarii Makarenko [ 29/May/19 ]

Facing same issue

"Message": "An exception occurred while receiving a message from the server.", "Source": "MongoDB.Driver.Core", "StackTraceString": " at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBufferAsync()\n at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBufferAsync(Int32 responseTo, CancellationToken cancellationToken)\n at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveMessageAsync(Int32 responseTo, IMessageEncoderSelector encoderSelector, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)\n at MongoDB.Driver.Core.WireProtocol.CommandUsingCommandMessageWireProtocol`1.ExecuteAsync(IConnection connection, CancellationToken cancellationToken)\n at MongoDB.Driver.Core.Servers.Server.ServerChannel.ExecuteProtocolAsync[TResult](IWireProtocol`1 protocol, CancellationToken cancellationToken)\n at MongoDB.Driver.Core.Operations.CommandOperationBase`1.ExecuteProtocolAsync(IChannelSource channelSource, ICoreSessionHandle session, ReadPreference readPreference, CancellationToken cancellationToken)\n at MongoDB.Driver.Core.Operations.ReadCommandOperation`1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)\n at MongoDB.Driver.Core.Operations.FindCommandOperation`1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)\n at MongoDB.Driver.Core.Operations.FindOperation`1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)\n at MongoDB.Driver.OperationExecutor.ExecuteReadOperationAsync[TResult](IReadBinding binding, IReadOperation`1 operation, CancellationToken cancellationToken)\n at MongoDB.Driver.MongoCollectionImpl`1.ExecuteReadOperationAsync[TResult](IClientSessionHandle session, IReadOperation`1 operation, ReadPreference readPreference, CancellationToken cancellationToken)\n at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)\n at MongoDB.Driver.IAsyncCursorSourceExtensions.FirstOrDefaultAsync[TDocument](IAsyncCursorSource`1 source, CancellationToken cancellationToken)\n at Global.Orchestrators.Infrastructure.StateManagement.MongoDbMachineStateProvider.FetchAsync(Guid id) in /source/src/BuildingBlocks/EventOrchestrator/StateManagement/MongoDbMachineStateProvider.cs:line 63\n at BuildingBlocks.EventOrchestrator.StateManagement.TransitionalMachineStateProvider.FetchAsync(Guid id) in /source/src/BuildingBlocks/EventOrchestrator/StateManagement/TransitionaMachineStateProvider.cs:line 34\n at RawRabbit.Operations.StateMachine.Core.StateMachineActivator.ActivateAsync(Guid id, Type stateMachineType)\n at RawRabbit.Operations.StateMachine.Middleware.RetrieveStateMachineMiddleware.InvokeAsync(IPipeContext context, CancellationToken token)\n at RawRabbit.Operations.StateMachine.Core.ProcessGlobalLock.ExecuteAsync(Guid modelId, Func`1 handler, CancellationToken ct)",

Comment by Mitereiter Balazs Zoltan [ 28/May/19 ]

Any idea on this matter?

Comment by Mitereiter Balazs Zoltan [ 20/Mar/19 ]

We acquired a new error message with the inner exception:

2019-03-20 03:06:51 MongoConnectionException: keywordId: 8721833; countryCode: D
 2019-03-20 03:06:51 An exception occurred while receiving a message from the server.
 MongoDB.Driver.Core
 ReceiveBuffer
 at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBuffer()
 at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBuffer(Int32 responseTo, CancellationToken cancellationToken)
 at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveMessage(Int32 responseTo, IMessageEncoderSelector encoderSelector, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)
 at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquiredConnection.ReceiveMessage(Int32 responseTo, IMessageEncoderSelector encoderSelector, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)
 at MongoDB.Driver.Core.WireProtocol.CommandUsingCommandMessageWireProtocol`1.Execute(IConnection connection, CancellationToken cancellationToken)
 at MongoDB.Driver.Core.WireProtocol.CommandWireProtocol`1.Execute(IConnection connection, CancellationToken cancellationToken)
 at MongoDB.Driver.Core.Servers.Server.ServerChannel.ExecuteProtocol[TResult](IWireProtocol`1 protocol, CancellationToken cancellationToken)
 at MongoDB.Driver.Core.Servers.Server.ServerChannel.Command[TResult](ICoreSession session, ReadPreference readPreference, DatabaseNamespace databaseNamespace, BsonDocument command, IEnumerable`1 commandPayloads, IElementNameValidator commandValidator, BsonDocument additionalOptions, Action`1 postWriteAction, CommandResponseHandling responseHandling, IBsonSerializer`1 resultSerializer, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)
 at MongoDB.Driver.Core.Operations.RetryableWriteCommandOperationBase.ExecuteAttempt(RetryableWriteContext context, Int32 attempt, Nullable`1 transactionNumber, CancellationToken cancellationToken)
 at MongoDB.Driver.Core.Operations.RetryableWriteOperationExecutor.Execute[TResult](IRetryableWriteOperation`1 operation, RetryableWriteContext context, CancellationToken cancellationToken)
 at MongoDB.Driver.Core.Operations.BulkUnmixedWriteOperationBase`1.ExecuteBatches(RetryableWriteContext context, CancellationToken cancellationToken)
 at MongoDB.Driver.Core.Operations.BulkUnmixedWriteOperationBase`1.Execute(RetryableWriteContext context, CancellationToken cancellationToken)
 at MongoDB.Driver.Core.Operations.BulkMixedWriteOperation.ExecuteBatch(RetryableWriteContext context, Batch batch, CancellationToken cancellationToken)
 at MongoDB.Driver.Core.Operations.BulkMixedWriteOperation.Execute(IWriteBinding binding, CancellationToken cancellationToken)
 at MongoDB.Driver.OperationExecutor.ExecuteWriteOperation[TResult](IWriteBinding binding, IWriteOperation`1 operation, CancellationToken cancellationToken)
 at MongoDB.Driver.MongoCollectionImpl`1.ExecuteWriteOperation[TResult](IClientSessionHandle session, IWriteOperation`1 operation, CancellationToken cancellationToken)
 at MongoDB.Driver.MongoCollectionImpl`1.BulkWrite(IClientSessionHandle session, IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)
 at MongoDB.Driver.MongoCollectionImpl`1.<>c__DisplayClass23_0.<BulkWrite>b__0(IClientSessionHandle session)
 at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSession[TResult](Func`2 func, CancellationToken cancellationToken)
 at MongoDB.Driver.MongoCollectionImpl`1.BulkWrite(IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)
 at MongoDB.Driver.MongoCollectionBase`1.<>c__DisplayClass82_0.<ReplaceOne>b__0(IEnumerable`1 requests, BulkWriteOptions bulkWriteOptions)
 at MongoDB.Driver.MongoCollectionBase`1.ReplaceOne(FilterDefinition`1 filter, TDocument replacement, UpdateOptions options, Func`3 bulkWrite)
 at MongoDB.Driver.MongoCollectionBase`1.ReplaceOne(FilterDefinition`1 filter, TDocument replacement, UpdateOptions options, CancellationToken cancellationToken)
 at ImportOracleToMongo.KeywordDAL.GetBaseDataTest(BackgroundWorker worker) in d:\Projects\NewProject\ImportOracleToMongo\ImportOracleToMongo\DAL\KeywordDAL.cs:line 194
 2019-03-20 03:06:54 InnerException: System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
 at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
 at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
 — End of inner exception stack trace —
 at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
 at MongoDB.Driver.Core.Misc.StreamExtensionMethods.ReadBytes(Stream stream, Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
 at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBuffer()

We are trying to migrate from Oracle to Mongo, with these tools we are trying to keep the 2 DBs at same state. We changed our cluster a little bit, we have now 2 data-bearing nodes + 1 arbiter in local network and the webserver (windows server 2016 Standard, Processor Intel(R) Xeon(R) CPU X5650 @ 2.67GHz, 2660 Mhz, 6 Core(s), 12 Logical Processor(s), Installed Physical Memory (RAM) 32.0 GB) where these tools are running is also in the same local network. Our connection string options at the moment are the following: replicaSet=shard1; readPreference=nearest; connectTimeoutMS=10000; socketTimeoutMS=10000; waitQueueMultiple=11; maxPoolSize=1111;

Comment by Ian Whalen (Inactive) [ 18/Mar/19 ]

mitereiter@gmail.com can you include the inner exception that should be included here and tell us exactly what happened?

Generated at Wed Feb 07 21:42:51 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.