[CSHARP-2645] SocketException: An operation on a socket could not be performed Created: 21/Jun/19  Updated: 22/Jul/19  Resolved: 22/Jul/19

Status: Closed
Project: C# Driver
Component/s: None
Affects Version/s: 2.8.1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Austin Felipe Assignee: Robert Stam
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Azure Win 10 (App services) - S1 1.75 GB
.Net Core 2.2



 Description   

Hi guys.

I'm running a stress test with JMeter and I'm facing the following error:

// Error
SocketException: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full
 
System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, string callerName)System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)System.Net.Sockets.Socket.InternalBind(EndPoint localEP)System.Net.Sockets.Socket.BeginConnectEx(EndPoint remoteEP, bool flowContext, AsyncCallback callback, object state)System.Net.Sockets.Socket.UnsafeBeginConnect(EndPoint remoteEP, AsyncCallback callback, object state, bool flowContext)System.Net.Sockets.Socket.BeginConnect(EndPoint remoteEP, AsyncCallback callback, object state)System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP)MongoDB.Driver.Core.Connections.TcpStreamFactory.ConnectAsync(Socket socket, EndPoint endPoint, CancellationToken cancellationToken)MongoDB.Driver.Core.Connections.TcpStreamFactory.CreateStreamAsync(EndPoint endPoint, CancellationToken cancellationToken)MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken)

 MongoConnectionException: An exception occurred while opening a connection to the server.
 
MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken)
MongoDB.Driver.Core.Servers.Server.GetChannelAsync(CancellationToken cancellationToken)
MongoDB.Driver.Core.Operations.FindOperation<TDocument>.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)
MongoDB.Driver.OperationExecutor.ExecuteReadOperationAsync<TResult>(IReadBinding binding, IReadOperation<TResult> operation, CancellationToken cancellationToken)
MongoDB.Driver.MongoCollectionImpl<TDocument>.ExecuteReadOperationAsync<TResult>(IClientSessionHandle session, IReadOperation<TResult> operation, ReadPreference readPreference, CancellationToken cancellationToken)
MongoDB.Driver.MongoCollectionImpl<TDocument>.UsingImplicitSessionAsync<TResult>(Func<IClientSessionHandle, Task<TResult>> funcAsync, CancellationToken cancellationToken)
MongoDB.Driver.IAsyncCursorSourceExtensions.FirstOrDefaultAsync<TDocument>(IAsyncCursorSource<TDocument> source, CancellationToken cancellationToken)

JMeter setup is:

100 threads in parallel
20x in loop (the next round starts after previews one has finished)

The error occurs around the 8th round.

It seems Windows is running out of ports to handle requests, but my question is: shouldn't sockets be reused?

In my head I thought that since the maximum parallel requests count is 100, it should use 100 ports only.

 

My Mongo client config (it's called once per application lifetime):

// code placeholder
var settings = MongoClientSettings.FromConnectionString(config.CacheConnection);void SocketConfigurator(Socket s) => s.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.KeepAlive, true);            settings.SocketTimeout = TimeSpan.FromMinutes(5);
settings.ConnectTimeout = TimeSpan.FromSeconds(60);
settings.MaxConnectionIdleTime = TimeSpan.FromSeconds(30);
settings.ClusterConfigurator = cb => cb.ConfigureTcp(tcp => tcp.With(socketConfigurator: (Action<Socket>)SocketConfigurator));            var mongoClient = new MongoClient(settings);            services.AddScoped<ICacheDatabase>(t => new CacheDatabase(config, mongoClient));

My socket config:

internal struct KeepAliveValues
        {
            public uint OnOff { get; set; }
            public uint KeepAliveTime { get; set; }
            public uint KeepAliveInterval { get; set; }
 
            public byte[] ToBytes()
            {
                var bytes = new byte[12];
                Array.Copy(BitConverter.GetBytes(OnOff), 0, bytes, 0, 4);
                Array.Copy(BitConverter.GetBytes(KeepAliveTime), 0, bytes, 4, 4);
                Array.Copy(BitConverter.GetBytes(KeepAliveInterval), 0, bytes, 8, 4);
                return bytes;
            }
        }
 
        internal static void SocketConfigurator(Socket s)
        {
            var keepAliveValues = new KeepAliveValues()
            {
                OnOff = 1,
                KeepAliveTime = 120 * 1000,   // 120 seconds in milliseconds
                KeepAliveInterval = 10 * 1000 // 10 seconds in milliseconds
            };
 
            s.IOControl(IOControlCode.KeepAliveValues, keepAliveValues.ToBytes(), null);
        }



 Comments   
Comment by Robert Stam [ 28/Jun/19 ]

Sorry I misinterpreted now many `MongoClient` instances you were using.

If you could provide a self contained `Program.cs` file that reproduces this that would be a tremendous help.

Comment by Austin Felipe [ 25/Jun/19 ]

Is there anything I could provide to help?

Comment by Austin Felipe [ 24/Jun/19 ]

Robert Stam, thanks for answering.

Well, actually I'm using one MongoClient per lifetime as I said on my post. That was a bad code formatting:

var settings = MongoClientSettings.FromConnectionString(config.CacheConnection);
 
void SocketConfigurator(Socket s) => s.SetSocketOption(SocketOptionLevel.Socket, 
    SocketOptionName.KeepAlive, true);
 
settings.SocketTimeout = TimeSpan.FromMinutes(5);
settings.ConnectTimeout = TimeSpan.FromSeconds(60);
settings.MaxConnectionIdleTime = TimeSpan.FromSeconds(30);
settings.ClusterConfigurator = cb => cb.ConfigureTcp(tcp => tcp.With(socketConfigurator: (Action<Socket>)SocketConfigurator));
 
// 1 mongo client per lifetime
var mongoClient = new MongoClient(settings);
 
services.AddScoped<ICacheDatabase>(t => new CacheDatabase(config, mongoClient));

That code is called once on Startup class. Each repository is using this CacheDatabase class which has:

private IMongoDatabase GetMongoDatabase(string databaseName)
{
            try
            {
                return client.GetDatabase(databaseName);
            }
            catch
            {
                Thread.Sleep(1000);                
                return GetMongoDatabase(databaseName);
            }
}

Any idea?

Comment by Robert Stam [ 24/Jun/19 ]

Normally when you create a new instance of `MongoClient` it shares an underlying connection pool with all other `MongoClient`  instances that have the same settings.

It looks like because you are using a `SocketConfigurator` the driver is unable to tell that all the `MongoClient` instances are compatible, in which case each new `MongoClient` instance would create a new connection pool. 

My suggestion would be to create a single global `MongoClient` instance and have all your threads share it.

Comment by Austin Felipe [ 22/Jun/19 ]

Code formatting is wrong, sorry about that guys. I'm not sure I can edit it.

Generated at Wed Feb 07 21:43:07 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.