[CSHARP-1482] Transient error handling Created: 19/Nov/15  Updated: 05/Apr/19  Resolved: 27/Jan/16

Status: Closed
Project: C# Driver
Component/s: Connectivity
Affects Version/s: 2.1
Fix Version/s: None

Type: Task Priority: Critical - P2
Reporter: Roberto Pérez Assignee: Unassigned
Resolution: Done Votes: 0
Labels: question
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Server side: a machine hosted in Microsoft Azure running Ubuntu-14_04-LTS and MongoDB 3.0.6

Client side: A cloud service hosted in Microsoft Azure running Windows 2012 R2 and making calls to the DB using the .NET driver (2.1.0).

Problem: when requesting too much data we always have an exception like this: System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time or established connection failed because connected host has failed to respond.

This is happening when reading 10 documents (each document in a different request one after other) with a size close to 1KB.
**The same tests requesting 1000 documents of 450B does not throw this exception so we are a bit lost

We have thought of retrying the failing statements when this kind of error happens, however, it would not be very clean as we should surround our driver queries/updates in Func<T> blocks to provide a transient error handling.

Could it be achieve via a subclass/configuration of the driver?

Thank you



 Comments   
Comment by Craig Wilson [ 27/Jan/16 ]

Hi Roberto,

Yes, let's open a new issue and I'll close this one down.

Craig

Comment by Roberto Pérez [ 26/Jan/16 ]

Hello again,

I do not know if I should open a new issue or this should be okay but the important problem is not the one mention in the description, the main problem is the one that I wrote in the last comment.

Regards

Comment by Roberto Pérez [ 23/Nov/15 ]

Update: We have been able to reproduce it in another way (or maybe it is a different error).

We have a web application that queries a MongoDB database and as we have thousands of documents we paginate on mongo (I tell this but I do not think it matters) and if we do not interact with the web application for some minutes, then, when we start again the first request fails (like in a timeout) and the Mongo Driver throws this exception:

Type: MongoConnectionException,
StackTrace: at MongoDB.Driver.Linq.MongoQueryProviderImpl`1.Execute(Expression expression)
at MongoDB.Driver.Linq.MongoQueryProviderImpl`1.Execute[TResult](Expression expression)
at System.Linq.Queryable.Count[TSource](IQueryable`1 source)

Next requests go okay.

I hope this helps!

Regards

Comment by Roberto Pérez [ 20/Nov/15 ]

Hello again,

We have found a network problem so the answer would be "a) there are network issues unrelated to the driver and you are wanting to know how to perform retries". Something that should not happen very often but if happens we would like to have a retry.

Thank you very much, you work very hard and answer everything very quickly

Comment by Roberto Pérez [ 20/Nov/15 ]

internal class MyPototoes
    {
        public int Id { get; set; }
        public List<string> Seeds { get; set; }
    }
 
    public class Program
    {
        public static void Main()
        {
            const string database = "aether_inventory";
            const string connectionString = "mongodb://user:password@host:port/" + database;
            const string collectionName = "mypotatoes";
 
            var mongoClient = new MongoClient(connectionString);
            var mongoDatabase = mongoClient.GetDatabase(database);
            var mongoCollection = mongoDatabase.GetCollection<MyPototoes>(collectionName);
 
            mongoDatabase.DropCollectionAsync(collectionName).Wait();
 
            const int messageCount = 10;
            var random = new Random();
 
            Enumerable.Range(0, messageCount)
                .Select(index => new MyPototoes
                {
                    Id = index,
                    Seeds = Enumerable.Range(0, 100).Select(i => random.Next(0, 9999999).ToString()).ToList()
                })
                .ToList()
                .ForEach(mp =>
                {
                    var sw = Stopwatch.StartNew();
                    Console.WriteLine("About to insert   [{0}], JsonStringSize: [{1}]", mp.Id, JsonConvert.SerializeObject(mp).Length);
                    try
                    {
                        mongoCollection.InsertOneAsync(mp).Wait();
                        Console.WriteLine("        inserted  [{0}] in [{1}] millis\n", mp.Id, sw.ElapsedMilliseconds);
                    }
                    catch (AggregateException ex)
                    {
                        Console.WriteLine("        failed    [{0}] in [{1}] millis\n", mp.Id, sw.ElapsedMilliseconds);
                        Console.WriteLine("Exception Info: {0}", ex.StackTrace);
 
                        throw;
                    }
                });
 
            Console.WriteLine();
            Console.WriteLine("All data inserted, now reading it");
            Console.WriteLine();
            
            for (var index = 0; index < messageCount; index++)
            {
                var sw = Stopwatch.StartNew();
                Console.WriteLine("About to read     [{0}]", index);
                try
                {
                    var responseDocumentTask = mongoCollection.AsQueryable().Where(d => d.Id == index).FirstOrDefaultAsync();
                    responseDocumentTask.Wait();
 
                    var responseDocument = responseDocumentTask.Result;
 
                    Console.WriteLine("        read      [{0}] in [{1}] millis, JsonStringSize: [{2}]\n", index, sw.ElapsedMilliseconds, JsonConvert.SerializeObject(responseDocument).Length);
                }
                catch (AggregateException ex)
                {
                    Console.WriteLine("        failed    [{0}] in [{1}] millis\n", index, sw.ElapsedMilliseconds);
                    Console.WriteLine("Exception Info: {0}", ex.StackTrace);
 
                    throw;
                }
            }
        }
    }

Execution output:

About to insert   [0], JsonStringSize: [1003]
        inserted  [0] in [181] millis
 
About to insert   [1], JsonStringSize: [1009]
        inserted  [1] in [34] millis
 
About to insert   [2], JsonStringSize: [1004]
        inserted  [2] in [34] millis
 
About to insert   [3], JsonStringSize: [1001]
        inserted  [3] in [34] millis
 
About to insert   [4], JsonStringSize: [1005]
        inserted  [4] in [35] millis
 
About to insert   [5], JsonStringSize: [1005]
        inserted  [5] in [33] millis
 
About to insert   [6], JsonStringSize: [1006]
        inserted  [6] in [34] millis
 
About to insert   [7], JsonStringSize: [1009]
        inserted  [7] in [36] millis
 
About to insert   [8], JsonStringSize: [1002]
        inserted  [8] in [34] millis
 
About to insert   [9], JsonStringSize: [1006]
        failed    [9] in [18916] millis
 
Exception Info:    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at ConsoleMono.Program.<>c__DisplayClass4.<Main>b__1(MyPototoes mp) in d:\tfs2013\RMM\Server\Main\AetherServer\ConsoleMono\Program.cs:line 49
 
Unhandled Exception: System.AggregateException: One or more errors occurred. ---> MongoDB.Driver.MongoConnectionException: An exception occurred while receiving a message from the server. ---> System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time or established connection failed because connected host has failed to respond
   at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
   at System.IO.Stream.<BeginEndReadAsync>b__e(Stream stream, IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncTrimPromise`1.Complete(TInstance thisRef, Func`3 endMethod, IAsyncResult asyncResult, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MongoDB.Driver.Core.Misc.StreamExtensionMethods.<ReadBytesAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MongoDB.Driver.Core.Connections.BinaryConnection.<ReceiveBufferAsync>d__45.MoveNext()
   --- End of inner exception stack trace ---
   at MongoDB.Driver.Core.Connections.BinaryConnection.<ReceiveBufferAsync>d__45.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MongoDB.Driver.Core.Connections.BinaryConnection.<ReceiveBufferAsync>d__46.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MongoDB.Driver.Core.Connections.BinaryConnection.<ReceiveMessageAsync>d__47.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MongoDB.Driver.Core.WireProtocol.CommandWireProtocol`1.<ExecuteAsync>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MongoDB.Driver.Core.Servers.ClusterableServer.ServerChannel.<ExecuteProtocolAsync>d__15`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MongoDB.Driver.Core.Operations.BulkUnmixedWriteOperationBase.<ExecuteBatchAsync>d__38.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MongoDB.Driver.Core.Operations.BulkUnmixedWriteOperationBase.<ExecuteBatchesAsync>d__39.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MongoDB.Driver.Core.Operations.BulkUnmixedWriteOperationBase.<ExecuteAsync>d__36.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MongoDB.Driver.Core.Operations.BulkMixedWriteOperation.<ExecuteBatchAsync>d__36.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MongoDB.Driver.Core.Operations.BulkMixedWriteOperation.<ExecuteAsync>d__34.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MongoDB.Driver.Core.Operations.BulkMixedWriteOperation.<ExecuteAsync>d__35.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MongoDB.Driver.OperationExecutor.<ExecuteWriteOperationAsync>d__1`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MongoDB.Driver.MongoCollectionImpl`1.<ExecuteWriteOperationAsync>d__35`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MongoDB.Driver.MongoCollectionImpl`1.<BulkWriteAsync>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MongoDB.Driver.MongoCollectionBase`1.<InsertOneAsync>d__20.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at ConsoleMono.Program.<>c__DisplayClass4.<Main>b__1(MyPototoes mp) in d:\tfs2013\RMM\Server\Main\AetherServer\ConsoleMono\Program.cs:line 57
   at System.Collections.Generic.List`1.ForEach(Action`1 action)
   at ConsoleMono.Program.Main() in d:\tfs2013\RMM\Server\Main\AetherServer\ConsoleMono\Program.cs:line 36
Press any key to continue . . .

Comment by Craig Wilson [ 19/Nov/15 ]

Just for clarity, are you saying

a) there are network issues unrelated to the driver and you are wanting to know how to perform retries
b) the driver is failing and you'd like us to fix it.

If (b), could you provide the code you are using or a simple repro?

Thanks,
Craig

Generated at Wed Feb 07 21:39:43 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.