[CSHARP-71] Running a FindAll which returns more than 16KB of results causes an error. Created: 16/Oct/10  Updated: 16/Nov/21  Resolved: 18/Oct/10

Status: Closed
Project: C# Driver
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Barry Dahlberg Assignee: Robert Stam
Resolution: Done Votes: 3
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

IIS 7, .Net 4, Windows 7 64 bit.


Attachments: File csharp71.bson     Text File json-with-docarray.txt    

 Description   

This works fine with result sets up to around 16KB. I notice that BsonBuffer.chunkSize is set to 16KB, seems likely to be related? See the following stack trace issued when trying to create a new list from a FindAll query:

[EndOfStreamException: Not enough input bytes available]
MongoDB.BsonLibrary.IO.BsonBuffer.EnsureDataAvailable(Int32 needed) +159
MongoDB.BsonLibrary.IO.BsonBuffer.ReadString() +217
MongoDB.BsonLibrary.IO.BsonBinaryReader.ReadString(String& name) +299
MongoDB.BsonLibrary.BsonElement.ReadFrom(BsonReader bsonReader, BsonElement& element) +422
MongoDB.BsonLibrary.BsonDocument.Deserialize(BsonReader bsonReader) +111
MongoDB.BsonLibrary.BsonDocument.ReadFrom(BsonReader bsonReader) +119
MongoDB.BsonLibrary.BsonElement.ReadFrom(BsonReader bsonReader, BsonElement& element) +550
MongoDB.BsonLibrary.BsonDocument.Deserialize(BsonReader bsonReader) +111
MongoDB.BsonLibrary.BsonDocument.ReadFrom(BsonReader bsonReader) +119
MongoDB.BsonLibrary.BsonElement.ReadFrom(BsonReader bsonReader, BsonElement& element) +550
MongoDB.BsonLibrary.BsonDocument.Deserialize(BsonReader bsonReader) +111
MongoDB.BsonLibrary.BsonDocument.ReadFrom(BsonReader bsonReader) +119
MongoDB.BsonLibrary.Serialization.BsonSerializer.Deserialize(BsonReader bsonReader, Type type) +130
MongoDB.BsonLibrary.Serialization.BsonSerializer.Deserialize(BsonReader bsonReader) +110
MongoDB.CSharpDriver.Internal.MongoReplyMessage`1.ReadFrom(BsonBuffer buffer) +1119
MongoDB.CSharpDriver.Internal.MongoConnection.ReceiveMessage() +470
MongoDB.CSharpDriver.MongoCursorEnumerator.GetReply(MongoRequestMessage message) +262
MongoDB.CSharpDriver.MongoCursorEnumerator.GetFirst() +937
MongoDB.CSharpDriver.MongoCursorEnumerator.MoveNext() +260
System.Collections.Generic.List`1..ctor(IEnumerable`1 collection) +472



 Comments   
Comment by Christian Sparre [ 18/Oct/10 ]

Robert, the fix did it. Thank you

Must get myself a git client, downloading those .zip files gets old very fast!!

Comment by Robert Stam [ 18/Oct/10 ]

Barry, master should be in a usable state. It is passing all the unit tests. I try to keep master continuously in a usable state although obviously sometimes that won't be true!

Comment by Robert Stam [ 18/Oct/10 ]

Christian, thanks for the file. I think I found the cause of the problem (see previous comment). An absurdly large value for size is a symptom that the reader lost its position in the buffers and read some random 4 bytes from the wrong place. A document can straddle a buffer chunk boundary either by being very big (as I used to reproduce the issue) or by being part of a large number of documents returned by FindAll (which is what you are seeing).

Thanks so much for helping me reproduce this. Let me know if the fix committed does or does not resolve your exception.

Comment by Barry Dahlberg [ 18/Oct/10 ]

Thanks, glad you got that one, I had been unable to make it happen again, obviously my documents had a subtle change.

Is master in a usable state?

Comment by Robert Stam [ 18/Oct/10 ]

Fixed in master. Bug occurs when an element name in a large document straddles a chunk boundary in the BsonBuffer.

Comment by Christian Sparre [ 18/Oct/10 ]

Hi Robert

I've attached a .bson file like you asked. The collection contains 63 of those and they are all around the same size +/- a few bytes.

I'm getting the following exception

Message:
Not enough input bytes available: 1953723716 needed but only 979 available (at position 32779)

Stacktrace:
at MongoDB.BsonLibrary.IO.BsonBuffer.EnsureDataAvailable(Int32 needed) in D:\Projekter\mongodb-mongo-csharp-driver-5994ea3\BsonLibrary\IO\BsonBuffer.cs:line 590
at MongoDB.BsonLibrary.IO.BsonBuffer.ReadString() in D:\Projekter\mongodb-mongo-csharp-driver-5994ea3\BsonLibrary\IO\BsonBuffer.cs:line 340
at MongoDB.BsonLibrary.IO.BsonBinaryReader.ReadString(String& name) in D:\Projekter\mongodb-mongo-csharp-driver-5994ea3\BsonLibrary\IO\BsonBinaryReader.cs:line 481
at MongoDB.BsonLibrary.BsonElement.ReadFrom(BsonReader bsonReader, BsonElement& element) in D:\Projekter\mongodb-mongo-csharp-driver-5994ea3\BsonLibrary\ObjectModel\BsonElement.cs:line 117
at MongoDB.BsonLibrary.BsonDocument.DeserializeDocument(BsonReader bsonReader, Type nominalType) in D:\Projekter\mongodb-mongo-csharp-driver-5994ea3\BsonLibrary\ObjectModel\BsonDocument.cs:line 380
at MongoDB.BsonLibrary.BsonDocument.ReadFrom(BsonReader bsonReader) in D:\Projekter\mongodb-mongo-csharp-driver-5994ea3\BsonLibrary\ObjectModel\BsonDocument.cs:line 214
at MongoDB.BsonLibrary.BsonElement.ReadFrom(BsonReader bsonReader, BsonElement& element) in D:\Projekter\mongodb-mongo-csharp-driver-5994ea3\BsonLibrary\ObjectModel\BsonElement.cs:line 121
at MongoDB.BsonLibrary.BsonArray.ReadFrom(BsonReader bsonReader) in D:\Projekter\mongodb-mongo-csharp-driver-5994ea3\BsonLibrary\ObjectModel\BsonArray.cs:line 218
at MongoDB.BsonLibrary.BsonElement.ReadFrom(BsonReader bsonReader, BsonElement& element) in D:\Projekter\mongodb-mongo-csharp-driver-5994ea3\BsonLibrary\ObjectModel\BsonElement.cs:line 125
at MongoDB.BsonLibrary.BsonDocument.DeserializeDocument(BsonReader bsonReader, Type nominalType) in D:\Projekter\mongodb-mongo-csharp-driver-5994ea3\BsonLibrary\ObjectModel\BsonDocument.cs:line 380
at MongoDB.BsonLibrary.BsonDocument.ReadFrom(BsonReader bsonReader) in D:\Projekter\mongodb-mongo-csharp-driver-5994ea3\BsonLibrary\ObjectModel\BsonDocument.cs:line 214
at MongoDB.BsonLibrary.Serialization.BsonSerializer.DeserializeDocument(BsonReader bsonReader, Type nominalType) in D:\Projekter\mongodb-mongo-csharp-driver-5994ea3\BsonLibrary\Serialization\BsonSerializer.cs:line 64
at MongoDB.BsonLibrary.Serialization.BsonSerializer.DeserializeDocument[T](BsonReader bsonReader) in D:\Projekter\mongodb-mongo-csharp-driver-5994ea3\BsonLibrary\Serialization\BsonSerializer.cs:line 44
at MongoDB.CSharpDriver.Internal.MongoReplyMessage`1.ReadFrom(BsonBuffer buffer) in D:\Projekter\mongodb-mongo-csharp-driver-5994ea3\CSharpDriver\Internal\MongoReplyMessage.cs:line 88
at MongoDB.CSharpDriver.Internal.MongoConnection.ReceiveMessage[TDocument]() in D:\Projekter\mongodb-mongo-csharp-driver-5994ea3\CSharpDriver\Internal\MongoConnection.cs:line 291
at MongoDB.CSharpDriver.MongoCursor`2.MongoCursorEnumerator.GetReply(MongoRequestMessage message) in D:\Projekter\mongodb-mongo-csharp-driver-5994ea3\CSharpDriver\Core\MongoCursor.cs:line 477
at MongoDB.CSharpDriver.MongoCursor`2.MongoCursorEnumerator.GetFirst() in D:\Projekter\mongodb-mongo-csharp-driver-5994ea3\CSharpDriver\Core\MongoCursor.cs:line 442
at MongoDB.CSharpDriver.MongoCursor`2.MongoCursorEnumerator.MoveNext() in D:\Projekter\mongodb-mongo-csharp-driver-5994ea3\CSharpDriver\Core\MongoCursor.cs:line 369
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at ConsoleApplication2.Program.Main(String[] args) in c:\users\christian\documents\visual studio 2010\Projects\ConsoleApplication2\ConsoleApplication2\Program.cs:line 16
at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()

I also tried to skip my way into the collection but ended up with the same result. This is some of the exception messages

Skip(20) Limit(5)
Not enough input bytes available: 1836410965 needed but only 414 available (at position 16395)

Skip(20) Limit(10)
Not enough input bytes available: 1836410965 needed but only 17225 available (at position 16395)

When I'm at the "start" of the collection I can get 9 documents, but when I'm "inside" the collection I can only get fewer.

Also if 1836410965 is bytes, that is quite a lot

I hope the above messages and info helps.

Comment by Robert Stam [ 18/Oct/10 ]

I've also enhanced the error message to indicate how many bytes were needed, how many were available, and to report the position in the buffer(s) at which the error occurred.

Comment by Robert Stam [ 18/Oct/10 ]

I am unable to reproduce this using the "json-with-docarray.txt" file attached, but I think it's because some of the values are

{ "Value", "System.Byte[]" }

and the value is not really a string but a much larger binary value. I'll keep investigating to see if I can reproduce independently. If one of you could instrument your program to output the binary bson form that would help me a lot. Add:

document.WriteTo("csharp71.bson"); // assuming document is of type BsonDocument

to your program or set a breakpoint and execute the above statement in the immediate window and attach the "csharp71.bson" file to this bug report.

Thanks!

Comment by Barry Dahlberg [ 17/Oct/10 ]

I have embedded documents but no arrays.

Comment by Christian Sparre [ 17/Oct/10 ]

On further investigation I seem to only get this on documents that have embedded arrays of documents. I have attached "json-with-docarray.txt" with an example of one of the documents that causes the error.

I have tested on documents with byte arrays in the 1-2 MB size, and there is no problems.

Comment by Christian Sparre [ 16/Oct/10 ]

I'm seeing this on the Find method as well

Generated at Wed Feb 07 21:35:43 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.