[CSHARP-1254] Unable to serialize extremely large volume of data (>2.8GB) Created: 24/Apr/15  Updated: 20/Jan/16  Resolved: 20/May/15

Status: Closed
Project: C# Driver
Component/s: BSON
Affects Version/s: 1.3.1, 2.0
Fix Version/s: 2.1

Type: Bug Priority: Minor - P4
Reporter: Chris Alexander Assignee: Robert Stam
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows 7 Enterprise SP1 x64 (and probably others)


Attachments: Zip Archive BsonDataVolume.zip    

 Description   

I attempted to serialize over 2.8GB of data in one go using a stream to a file on my disk, and was unable to complete the operation due to the below exception. I was originally using a pretty old version of the driver and ran into a similar integer overflow issue, so when I upgraded I decided to check whether it also affects this version, which it does.

The root cause seems to be that I am trying to serialize more bytes than the Position data type can track, resulting in an overflow. I have also attached a very simple project which reproduces the issue (on my machine at least).

I'm sure that the workaround provided will be to split the data up but I noticed the implementation changed since my previous version to use streams rather than buffers so I had hoped this would have solved the issue.

Thank you!

Full exception:
System.ArgumentOutOfRangeException was unhandled
HResult=-2146233086
Message=Non-negative number required.
Parameter name: value
Source=mscorlib
ParamName=value
StackTrace:
at System.IO.FileStream.set_Position(Int64 value)
at MongoDB.Bson.IO.BsonStreamAdapter.set_Position(Int64 value) in d:\jenkins\workspace\mongo-csharp-driver-build\src\MongoDB.Bson\IO\BsonStreamAdapter.cs:line 132
at MongoDB.Bson.IO.BsonBinaryWriter.BackpatchSize() in d:\jenkins\workspace\mongo-csharp-driver-build\src\MongoDB.Bson\IO\BsonBinaryWriter.cs:line 717
at MongoDB.Bson.IO.BsonBinaryWriter.WriteEndArray() in d:\jenkins\workspace\mongo-csharp-driver-build\src\MongoDB.Bson\IO\BsonBinaryWriter.cs:line 309
at MongoDB.Bson.Serialization.Serializers.EnumerableSerializerBase`2.Serialize(BsonSerializationContext context, BsonSerializationArgs args, TValue value) in d:\jenkins\workspace\mongo-csharp-driver-build\src\MongoDB.Bson\Serialization\Serializers\EnumerableSerializerBase.cs:line 317
at MongoDB.Bson.Serialization.IBsonSerializerExtensions.Serialize[TValue](IBsonSerializer`1 serializer, BsonSerializationContext context, TValue value) in d:\jenkins\workspace\mongo-csharp-driver-build\src\MongoDB.Bson\Serialization\IBsonSerializerExtensions.cs:line 74
at MongoDB.Bson.Serialization.Serializers.EnumerableSerializerBase`2.Serialize(BsonSerializationContext context, BsonSerializationArgs args, TValue value) in d:\jenkins\workspace\mongo-csharp-driver-build\src\MongoDB.Bson\Serialization\Serializers\EnumerableSerializerBase.cs:line 313
at MongoDB.Bson.Serialization.Serializers.SerializerBase`1.MongoDB.Bson.Serialization.IBsonSerializer.Serialize(BsonSerializationContext context, BsonSerializationArgs args, Object value) in d:\jenkins\workspace\mongo-csharp-driver-build\src\MongoDB.Bson\Serialization\Serializers\SerializerBase.cs:line 125
at MongoDB.Bson.Serialization.IBsonSerializerExtensions.Serialize(IBsonSerializer serializer, BsonSerializationContext context, Object value) in d:\jenkins\workspace\mongo-csharp-driver-build\src\MongoDB.Bson\Serialization\IBsonSerializerExtensions.cs:line 61
at MongoDB.Bson.Serialization.BsonClassMapSerializer`1.SerializeNormalMember(BsonSerializationContext context, Object obj, BsonMemberMap memberMap) in d:\jenkins\workspace\mongo-csharp-driver-build\src\MongoDB.Bson\Serialization\Serializers\BsonClassMapSerializer.cs:line 594
at MongoDB.Bson.Serialization.BsonClassMapSerializer`1.SerializeMember(BsonSerializationContext context, Object obj, BsonMemberMap memberMap) in d:\jenkins\workspace\mongo-csharp-driver-build\src\MongoDB.Bson\Serialization\Serializers\BsonClassMapSerializer.cs:line 573
at MongoDB.Bson.Serialization.BsonClassMapSerializer`1.SerializeClass(BsonSerializationContext context, BsonSerializationArgs args, TClass document) in d:\jenkins\workspace\mongo-csharp-driver-build\src\MongoDB.Bson\Serialization\Serializers\BsonClassMapSerializer.cs:line 516
at MongoDB.Bson.Serialization.BsonClassMapSerializer`1.Serialize(BsonSerializationContext context, BsonSerializationArgs args, TClass value) in d:\jenkins\workspace\mongo-csharp-driver-build\src\MongoDB.Bson\Serialization\Serializers\BsonClassMapSerializer.cs:line 350
at MongoDB.Bson.Serialization.Serializers.SerializerBase`1.MongoDB.Bson.Serialization.IBsonSerializer.Serialize(BsonSerializationContext context, BsonSerializationArgs args, Object value) in d:\jenkins\workspace\mongo-csharp-driver-build\src\MongoDB.Bson\Serialization\Serializers\SerializerBase.cs:line 125
at MongoDB.Bson.Serialization.BsonSerializer.Serialize(IBsonWriter bsonWriter, Type nominalType, Object value, Action`1 configurator, BsonSerializationArgs args) in d:\jenkins\workspace\mongo-csharp-driver-build\src\MongoDB.Bson\Serialization\BsonSerializer.cs:line 663
at BsonDataVolume.Program.Main(String[] args) in c:\Users\chris.alexander\Documents\Visual Studio 2013\Projects\BsonDataVolume\BsonDataVolume\Program.cs:line 43
at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
InnerException:



 Comments   
Comment by Githook User [ 20/May/15 ]

Author:

{u'username': u'rstam', u'name': u'rstam', u'email': u'robert@robertstam.org'}

Message: CSHARP-1254: Better error message when BackpatchSize is called and size is greater than 2GB.
Branch: master
https://github.com/mongodb/mongo-csharp-driver/commit/fac1e0efd95c3cd8c807cdc8a42059f73dddf58b

Comment by Robert Stam [ 15/May/15 ]

The maximum size of a BSON document is determined by the BSON spec (see: http://bsonspec.org/spec.html), which uses a signed 32-bit integer to represent the size of a document. Therefore, the largest possible BSON document is 2,147,483,647 bytes long (hex 0x7fffffff).

What we can do is give a better error message.

Comment by Craig Wilson [ 25/Apr/15 ]

Ok, we should be able to get this fixed. Hopefully it doesn't require public API changes... If it does, we'll have to wait until 3.0.

Comment by Chris Alexander [ 25/Apr/15 ]

Hi Craig

Sorry I forgot to mention, that is the case - just using Bson driver to handle a bit of serialisation for me.

Many thanks

Chris

Comment by Craig Wilson [ 24/Apr/15 ]

Hi Chris,

Just to be thorough there, you are using the Bson library by itself without the driver. Is that correct?

Craig

Generated at Wed Feb 07 21:39:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.