Uploaded image for project: 'C# Driver'
  1. C# Driver
  2. CSHARP-1254

Unable to serialize extremely large volume of data (>2.8GB)

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Minor - P4 Minor - P4
    • 2.1
    • Affects Version/s: 1.3.1, 2.0
    • Component/s: BSON
    • Labels:
      None
    • Environment:
      Windows 7 Enterprise SP1 x64 (and probably others)

      I attempted to serialize over 2.8GB of data in one go using a stream to a file on my disk, and was unable to complete the operation due to the below exception. I was originally using a pretty old version of the driver and ran into a similar integer overflow issue, so when I upgraded I decided to check whether it also affects this version, which it does.

      The root cause seems to be that I am trying to serialize more bytes than the Position data type can track, resulting in an overflow. I have also attached a very simple project which reproduces the issue (on my machine at least).

      I'm sure that the workaround provided will be to split the data up but I noticed the implementation changed since my previous version to use streams rather than buffers so I had hoped this would have solved the issue.

      Thank you!

      Full exception:
      System.ArgumentOutOfRangeException was unhandled
      HResult=-2146233086
      Message=Non-negative number required.
      Parameter name: value
      Source=mscorlib
      ParamName=value
      StackTrace:
      at System.IO.FileStream.set_Position(Int64 value)
      at MongoDB.Bson.IO.BsonStreamAdapter.set_Position(Int64 value) in d:\jenkins\workspace\mongo-csharp-driver-build\src\MongoDB.Bson\IO\BsonStreamAdapter.cs:line 132
      at MongoDB.Bson.IO.BsonBinaryWriter.BackpatchSize() in d:\jenkins\workspace\mongo-csharp-driver-build\src\MongoDB.Bson\IO\BsonBinaryWriter.cs:line 717
      at MongoDB.Bson.IO.BsonBinaryWriter.WriteEndArray() in d:\jenkins\workspace\mongo-csharp-driver-build\src\MongoDB.Bson\IO\BsonBinaryWriter.cs:line 309
      at MongoDB.Bson.Serialization.Serializers.EnumerableSerializerBase`2.Serialize(BsonSerializationContext context, BsonSerializationArgs args, TValue value) in d:\jenkins\workspace\mongo-csharp-driver-build\src\MongoDB.Bson\Serialization\Serializers\EnumerableSerializerBase.cs:line 317
      at MongoDB.Bson.Serialization.IBsonSerializerExtensions.Serialize[TValue](IBsonSerializer`1 serializer, BsonSerializationContext context, TValue value) in d:\jenkins\workspace\mongo-csharp-driver-build\src\MongoDB.Bson\Serialization\IBsonSerializerExtensions.cs:line 74
      at MongoDB.Bson.Serialization.Serializers.EnumerableSerializerBase`2.Serialize(BsonSerializationContext context, BsonSerializationArgs args, TValue value) in d:\jenkins\workspace\mongo-csharp-driver-build\src\MongoDB.Bson\Serialization\Serializers\EnumerableSerializerBase.cs:line 313
      at MongoDB.Bson.Serialization.Serializers.SerializerBase`1.MongoDB.Bson.Serialization.IBsonSerializer.Serialize(BsonSerializationContext context, BsonSerializationArgs args, Object value) in d:\jenkins\workspace\mongo-csharp-driver-build\src\MongoDB.Bson\Serialization\Serializers\SerializerBase.cs:line 125
      at MongoDB.Bson.Serialization.IBsonSerializerExtensions.Serialize(IBsonSerializer serializer, BsonSerializationContext context, Object value) in d:\jenkins\workspace\mongo-csharp-driver-build\src\MongoDB.Bson\Serialization\IBsonSerializerExtensions.cs:line 61
      at MongoDB.Bson.Serialization.BsonClassMapSerializer`1.SerializeNormalMember(BsonSerializationContext context, Object obj, BsonMemberMap memberMap) in d:\jenkins\workspace\mongo-csharp-driver-build\src\MongoDB.Bson\Serialization\Serializers\BsonClassMapSerializer.cs:line 594
      at MongoDB.Bson.Serialization.BsonClassMapSerializer`1.SerializeMember(BsonSerializationContext context, Object obj, BsonMemberMap memberMap) in d:\jenkins\workspace\mongo-csharp-driver-build\src\MongoDB.Bson\Serialization\Serializers\BsonClassMapSerializer.cs:line 573
      at MongoDB.Bson.Serialization.BsonClassMapSerializer`1.SerializeClass(BsonSerializationContext context, BsonSerializationArgs args, TClass document) in d:\jenkins\workspace\mongo-csharp-driver-build\src\MongoDB.Bson\Serialization\Serializers\BsonClassMapSerializer.cs:line 516
      at MongoDB.Bson.Serialization.BsonClassMapSerializer`1.Serialize(BsonSerializationContext context, BsonSerializationArgs args, TClass value) in d:\jenkins\workspace\mongo-csharp-driver-build\src\MongoDB.Bson\Serialization\Serializers\BsonClassMapSerializer.cs:line 350
      at MongoDB.Bson.Serialization.Serializers.SerializerBase`1.MongoDB.Bson.Serialization.IBsonSerializer.Serialize(BsonSerializationContext context, BsonSerializationArgs args, Object value) in d:\jenkins\workspace\mongo-csharp-driver-build\src\MongoDB.Bson\Serialization\Serializers\SerializerBase.cs:line 125
      at MongoDB.Bson.Serialization.BsonSerializer.Serialize(IBsonWriter bsonWriter, Type nominalType, Object value, Action`1 configurator, BsonSerializationArgs args) in d:\jenkins\workspace\mongo-csharp-driver-build\src\MongoDB.Bson\Serialization\BsonSerializer.cs:line 663
      at BsonDataVolume.Program.Main(String[] args) in c:\Users\chris.alexander\Documents\Visual Studio 2013\Projects\BsonDataVolume\BsonDataVolume\Program.cs:line 43
      at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
      at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
      at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
      at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
      at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
      at System.Threading.ThreadHelper.ThreadStart()
      InnerException:

            Assignee:
            robert@mongodb.com Robert Stam
            Reporter:
            chris_alexander Chris Alexander
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: