[CSHARP-2495] EndOfStreamException: Attempted to read past the end of the stream. Created: 30/Jan/19  Updated: 28/Sep/20  Resolved: 25/Jun/20

Status: Closed
Project: C# Driver
Component/s: Connectivity
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Jonas Kamsker Assignee: Wan Bachtiar
Resolution: Done Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

C#.NET, Mongodb ATLAS


Issue Links:
Related
related to CSHARP-3216 EndOfStreamException An exception occ... Closed

 Description   

When I am trying to run multiple asynchronous requests against the server, I get following error: 

  Name Value Type
ConnectionId Unknown macro: { { ServerId }

}

MongoDB.Driver.Core.Connections.ConnectionId
Data {System.Collections.ListDictionaryInternal}|System.Collections.IDictionary {System.Collections.ListDictionaryInternal}|
|▶|ErrorLabels|Count = 0|System.Collections.Generic.IReadOnlyList<string> {System.Collections.Generic.List<string>}|
| |HResult|-2146233088|int|
| |HelpLink|null|string|
|◢|InnerException|{"An exception occurred while receiving a message from the server."}|System.Exception {MongoDB.Driver.MongoConnectionException}|
| |▶ ConnectionId|{{ ServerId : { ClusterId : 1, EndPoint : "Unspecified/weird0-shard-00-00-h6hi1.mongodb.net:27017" }, LocalValue : 90 }}|MongoDB.Driver.Core.Connections.ConnectionId|
| |▶ Data|{System.Collections.ListDictionaryInternal}|System.Collections.IDictionary {System.Collections.ListDictionaryInternal}|
| |▶ ErrorLabels|Count = 0|System.Collections.Generic.IReadOnlyList<string> {System.Collections.Generic.List<string>}|
| |HResult|-2146233088|int|
| |HelpLink|null|string|
| |◢ InnerException|{"Attempted to read past the end of the stream."}|System.Exception {System.IO.EndOfStreamException}|
| |▶ Data|{System.Collections.ListDictionaryInternal}
System.Collections.IDictionary {System.Collections.ListDictionaryInternal}
  HResult -2147024858 int
  HelpLink null string
  ▶ InnerException null System.Exception
  Message "Attempted to read past the end of the stream." string
  Source "MongoDB.Driver.Core" string
  StackTrace " at MongoDB.Driver.Core.Misc.StreamExtensionMethods.<ReadBytesAsync>d_2.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at MongoDB.Driver.Core.Connections.BinaryConnection.<ReceiveBufferAsync>d_51.MoveNext()" string
  ▶ TargetSite {Void MoveNext()}|System.Reflection.MethodBase {System.Reflection.RuntimeMethodInfo}|
| |▶ Static members| | |
| |▶ Non-Public members| | |
| |Message|"An exception occurred while receiving a message from the server."|string|
| |Source|"MongoDB.Driver.Core"|string|
| |StackTrace|" at MongoDB.Driver.Core.Connections.BinaryConnection.<ReceiveBufferAsync>d_51.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown --\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at MongoDB.Driver.Core.Connections.BinaryConnection.<ReceiveBufferAsync>d52.MoveNext()\r\n-- End of stack trace from previous location where exception was thrown --\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at MongoDB.Driver.Core.Connections.BinaryConnection.<ReceiveMessageAsync>d54.MoveNext()\r\n-- End of stack trace from previous location where exception was thrown --\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at MongoDB.Driver.Core.WireProtocol.CommandUsingQueryMessageWireProtocol`1.<ExecuteAsync>d14.MoveNext()\r\n-- End of stack trace from previous location where exception was thrown --\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at MongoDB.Driver.Core.Connections.IsMasterHelper.<GetResultAsync>d5.MoveNext()\r\n-- End of stack trace from previous location where exception was thrown --\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at MongoDB.Driver.Core.Connections.ConnectionInitializer.<InitializeConnectionAsync>d3.MoveNext()\r\n-- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d_48.MoveNext()"|string|
| |▶ TargetSite|{Void MoveNext()}|System.Reflection.MethodBase {System.Reflection.RuntimeMethodInfo}|
| |▶ Static members| | |
| |▶ Non-Public members| | |
| |Message|"An exception occurred while opening a connection to the server."|string|
| |Source|"MongoDB.Driver.Core"|string|
| |StackTrace|" at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d_48.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown --\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at MongoDB.Driver.Core.Servers.Server.<GetChannelAsync>d30.MoveNext()\r\n-- End of stack trace from previous location where exception was thrown --\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at MongoDB.Driver.Core.Operations.AggregateOperation`1.<ExecuteAsync>d55.MoveNext()\r\n-- End of stack trace from previous location where exception was thrown --\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at MongoDB.Driver.OperationExecutor.<ExecuteReadOperationAsync>d3`1.MoveNext()\r\n-- End of stack trace from previous location where exception was thrown --\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at MongoDB.Driver.MongoCollectionImpl`1.<ExecuteReadOperationAsync>d93`1.MoveNext()\r\n-- End of stack trace from previous location where exception was thrown --\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at MongoDB.Driver.MongoCollectionImpl`1.<AggregateAsync>d22`1.MoveNext()\r\n-- End of stack trace from previous location where exception was thrown --\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at MongoDB.Driver.MongoCollectionImpl`1.<UsingImplicitSessionAsync>d101`1.MoveNext()\r\n-- End of stack trace from previous location where exception was thrown --\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at MongoDB.Driver.AsyncCursorHelper.<FirstOrDefaultAsync>d2`1.MoveNext()\r\n-- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()\r\n at Amazon.Cloud.DataAccess.Layer.VirtualFolderDal.<WalkPath>d_11.MoveNext() in C:\\Users\\Weirdo\\source\\repos\\AmazonExploiter\\Amazon.Cloud.DataAccessLayer
Layer
VirtualFolderDal.cs:line 137"|string|
|▶|TargetSite|{Void MoveNext()}
System.Reflection.MethodBase {System.Reflection.RuntimeMethodInfo}

 

 

Affected Library Code:

// Code-Platzhalter
        private async Task<(bool success, int lastPosition, MongoVirtualFolder folder)> WalkPath(string[] pathParts, VirtualFolder startFolder = null)
        {
            var folder = await ToMongoVirtualFolder(startFolder) ?? _rootFolder;            for (var i = 0; i < pathParts.Length; i++)
            {
                var part = pathParts[i];
// 
// The error occurs here:
//
                var newFolder = await Collection
                    .Where(x => x.Parent == folder.Id && x.Name == part)
                    .FirstOrDefaultAsync()
                    .ConfigureAwait(false);
                if (newFolder == null)
                {
                    return (false, i, folder);
                }                folder = newFolder;
            }
            return (true, pathParts.Length, folder);
        }        public Task<VirtualFolder> CreateFolder(string path, VirtualFolder startFolder = null)
        {
            var folderParts = path.Split(new char[] { '\\', '/' }, StringSplitOptions.RemoveEmptyEntries);
            return CreateFolder(folderParts, startFolder);
        }        public async Task<VirtualFolder> CreateFolder(string[] folderParts, VirtualFolder startFolder = null)
        {
            var (success, i, folder) = await WalkPath(folderParts, startFolder);
            if (!success)
            {
                int j = i;
                try
                {
                    for (j = i; j < folderParts.Length; j++)
                    {
                        var newFolder = new MongoVirtualFolder
                        {
                            Name = folderParts[j],
                            Parent = folder.Id,
                        };                        await Collection.InsertOneAsync(newFolder);
                        folder = newFolder;
                    }
                }
                catch (MongoWriteException exception) when (exception.InnerException is MongoBulkWriteException InnerException)
                {
                    if (InnerException.WriteErrors.Any(x => x.Category == ServerErrorCategory.DuplicateKey))
                    {
                        //Duplicate key
                        var searchArray = folderParts.AsEnumerable().Skip(i).Take(folderParts.Length - i).ToArray();
                        Console.WriteLine($"Concurrency warning! Retrying ({string.Join("/", folderParts)})");
                        return await CreateFolder(searchArray, folder);
                    }                    throw;
                }
                catch (Exception e)
                {
                    Console.WriteLine(e);
                    throw;
                }                folder.Folders = new List<VirtualFolderSlim>();
                folder.VirtualFiles = new List<VirtualFile>();
            }
            else
            {
                await LoadFolderContents(folder);
            }            return folder;
        }

Affected Usercode:

// Entrypoint:
async Task<Dictionary<string, (DirectoryInfo, VirtualFolder)>> CreateFolders1(VirtualFolder fold, DirectoryInfo dir)
            {
                var result = new Dictionary<string, (DirectoryInfo, VirtualFolder)>();
                var dirs = dir.EnumerateDirectories("*", SearchOption.TopDirectoryOnly);                //var subFolders = await dirs.Select(x => _virtualFolderDal.CreateFolder(x.Name, fold)).WhenAll();                var subFolders = await dirs.Select(x => ___CreateFolder(fold, x)).WhenAll();
                foreach (var virtualFolder in subFolders.SelectMany(x => x))
                {
                    result[virtualFolder.Key] = virtualFolder.Value;
                }                return result;                async Task<Dictionary<string, (DirectoryInfo, VirtualFolder)>> ___CreateFolder(VirtualFolder __fold, DirectoryInfo __dir)
                {
                    var subFolder = await _virtualFolderDal.CreateFolder(__dir.Name, __fold);
                    return await CreateFolders1(subFolder, __dir);
                }
            }

 

If you have any questions please dont hesitate to ask. 



 Comments   
Comment by Pritee Bhandari [ 27/Sep/20 ]

Hi Team,

Even I got above error but it occurs sometimes. But it is very critical from my project perspective.
We are using .net framework 4.6.1 and Nuget package for mongocsharpdriver, MongoDB.Bson,MongoDB.Driver -  Version is 2.8.1 .

Following is the exception logs:

MongoDB.Driver.MongoConnectionException:
at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBuffer (MongoDB.Driver.Core, Version=2.8.1.0, Culture=neutral, PublicKeyToken=null)
at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBuffer (MongoDB.Driver.Core, Version=2.8.1.0, Culture=neutral, PublicKeyToken=null)
at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveMessage (MongoDB.Driver.Core, Version=2.8.1.0, Culture=neutral, PublicKeyToken=null)
at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool+AcquiredConnection.ReceiveMessage (MongoDB.Driver.Core, Version=2.8.1.0, Culture=neutral, PublicKeyToken=null)
at MongoDB.Driver.Core.WireProtocol.CommandUsingQueryMessageWireProtocol`1.Execute (MongoDB.Driver.Core, Version=2.8.1.0, Culture=neutral, PublicKeyToken=null)
at MongoDB.Driver.Core.WireProtocol.CommandWireProtocol`1.Execute (MongoDB.Driver.Core, Version=2.8.1.0, Culture=neutral, PublicKeyToken=null)
at MongoDB.Driver.Core.Servers.Server+ServerChannel.ExecuteProtocol (MongoDB.Driver.Core, Version=2.8.1.0, Culture=neutral, PublicKeyToken=null)
at MongoDB.Driver.Core.Servers.Server+ServerChannel.Command (MongoDB.Driver.Core, Version=2.8.1.0, Culture=neutral, PublicKeyToken=null)
at MongoDB.Driver.Core.Operations.CommandOperationBase`1.ExecuteProtocol (MongoDB.Driver.Core, Version=2.8.1.0, Culture=neutral, PublicKeyToken=null)
at MongoDB.Driver.Core.Operations.CommandOperationBase`1.ExecuteProtocol (MongoDB.Driver.Core, Version=2.8.1.0, Culture=neutral, PublicKeyToken=null)
at MongoDB.Driver.Core.Operations.ReadCommandOperation`1.Execute (MongoDB.Driver.Core, Version=2.8.1.0, Culture=neutral, PublicKeyToken=null)
at MongoDB.Driver.Core.Operations.FindCommandOperation`1.Execute (MongoDB.Driver.Core, Version=2.8.1.0, Culture=neutral, PublicKeyToken=null)
at MongoDB.Driver.Core.Operations.FindOperation`1.Execute (MongoDB.Driver.Core, Version=2.8.1.0, Culture=neutral, PublicKeyToken=null)
at MongoDB.Driver.OperationExecutor.ExecuteReadOperation (MongoDB.Driver, Version=2.8.1.0, Culture=neutral, PublicKeyToken=null)
at MongoDB.Driver.MongoCollectionImpl`1.ExecuteReadOperation (MongoDB.Driver, Version=2.8.1.0, Culture=neutral, PublicKeyToken=null)
at MongoDB.Driver.MongoCollectionImpl`1.ExecuteReadOperation (MongoDB.Driver, Version=2.8.1.0, Culture=neutral, PublicKeyToken=null)
at MongoDB.Driver.MongoCollectionImpl`1.FindSync (MongoDB.Driver, Version=2.8.1.0, Culture=neutral, PublicKeyToken=null)
at MongoDB.Driver.MongoCollectionImpl`1+<>c_DisplayClass41_0`1.<FindSync>b_0 (MongoDB.Driver, Version=2.8.1.0, Culture=neutral, PublicKeyToken=null)
at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSession (MongoDB.Driver, Version=2.8.1.0, Culture=neutral, PublicKeyToken=null)
at MongoDB.Driver.MongoCollectionImpl`1.FindSync (MongoDB.Driver, Version=2.8.1.0, Culture=neutral, PublicKeyToken=null)
at MongoDB.Driver.FindFluent`2.ToCursor (MongoDB.Driver, Version=2.8.1.0, Culture=neutral, PublicKeyToken=null)
at MongoDB.Driver.IAsyncCursorSourceExtensions.FirstOrDefault (MongoDB.Driver.Core, Version=2.8.1.0, Culture=neutral, PublicKeyToken=null)
at MongoDB.Driver.IFindFluentExtensions.FirstOrDefault (MongoDB.Driver, Version=2.8.1.0, Culture=neutral, PublicKeyToken=null)

Inner exception System.IO.EndOfStreamException handled at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBuffer:
at MongoDB.Driver.Core.Misc.StreamExtensionMethods.ReadBytes (MongoDB.Driver.Core, Version=2.8.1.0, Culture=neutral, PublicKeyToken=null)
at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBuffer (MongoDB.Driver.Core, Version=2.8.1.0, Culture=neutral, PublicKeyToken=null)

 

Please help me on this asap.

Comment by Rachelle Palmer [ 25/Jun/20 ]

Hi there, thank you for reaching out to MongoDB. As this sounds more like a support issue, I wanted to give you some resources to get this question answered more quickly:

  • our MongoDB support portal, located at support.mongodb.com
  • our MongoDB community forums, located here
  • If you are an Atlas customer, there is free support offered 24/7 in the lower right hand corner of the UI.

Thank you!
Rachelle

Comment by Adam Friedland [ 17/Feb/20 ]

Is there any update regarding this specific issue? I'm struggling with this exact problem which I seem to only encounter when deploying to an Azure Kubernetes Service. I've followed the production guidelines and set the net.ipv4.tcp_keepalive_time to 120 on both the database pod and the API querying from it. This didn't seem to help at all. I'm using a single dedicated instance as well, nothing fancy with replica sets.

 

Update: For anyone experiencing the same mind-numbing issue that I have, it appears that it was due to having mongo-express running on the same cluster. I can only assume it was holding random connections to my mongo container.

Comment by Olivier Quirion [ 18/Apr/19 ]

I'm getting this error while reading from oplog and replicaset changes primary server.

Here's my code :

var opLog = LocalDatabase.GetCollection<BsonDocument>("oplog.rs");
// We rebuild the filter every time we loop because any 'second attemp' is the consequence of a connection exception.
FilterDefinition<BsonDocument> filter = buildFilter(lastInsertDate);                    
 
using (var cursor = opLog.FindSync(filter, options, CancellationToken)) {
   ConnectionIsAlive = true;                        
   //foreach (var document in cursor.ToList())
   foreach (var document in cursor.ToEnumerable()) {
         // Set the last value we saw 
         lastInsertDate = document["ts"].AsBsonTimestamp;
         ObjectId updatedObjectId = document["o"]["_id"].AsObjectId;
         string op = document["op"].ToString();
         PersistedObject persistedObject;
         log.InfoFormat("Object with id {0} has changed 'op'==> {1}.", updatedObjectId, op);                            
 
         //... some other logic
   }
}

MongoDB .Net Driver version : 2.7.2

MongoDB Server : 4.0.4

Regards,

Comment by Jonas Kamsker [ 18/Apr/19 ]

Hey Wan,

currently im not able to reproduce this as im little in time but i will get this done asap.

What i can say for now is, that i was querying the database recursively. That means that one query resulted in multiple other queries i executed and so on.. On the end there were several thousand queries at a time. One possible solution would be to allow the user to set a artificial rate limit (per [milli-]second/ timespan). I used https://github.com/David-Desmaisons/RateLimiter to accomplish this

Comment by Wan Bachtiar [ 12/Apr/19 ]

Hi,

Generally this is an exception that's caused by the lost of an expected server's response.
However we need more information to be able to reproduce the issue and confirm that this is not the driver's issue. Are you able to consistently reproduce the issue ?
Could you answer the following:

  • Which version of MongoDB Atlas that you're connecting to ?
  • Which version of MongoDB C# driver that you're using ?
  • Can you provide a minimal and complete code that is able to consistently trigger the exception ? i.e. without the folders related code.
  • Can you see any unusual messages in the server log around the exception time ? i.e. replica set election or fail-over

Regards,
Wan.

Generated at Wed Feb 07 21:42:43 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.