Uploaded image for project: 'C# Driver'
  1. C# Driver
  2. CSHARP-1626

Reading and writing Streams with more than 2GB of BSON data

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Minor - P4 Minor - P4
    • 2.2.4
    • Affects Version/s: 2.2.3
    • Component/s: BSON, Serialization
    • None
    • Environment:
      C#, .NET 4.6.1

      I reading a BSON dump file in order to convert the object from one format to another. When attempting to write the converted object out, I'm getting the an error message from MongoDB.Bson.IO.BSonBinaryWriter.BackpatchSize. The message is "Size 4294967329 is larger than MaxDocumentSize 2147483647.".

      Below is the code in question. EanSet and ActiveRecommendations are the two objects involved. EanSet is the object that is in Bson dump file. ActiveRecommendations is the new object. ActiveRecommendations should be a smaller object than EanSet.

      I know that my source data should be greater than 2MB.

      My goal is to take the new BSON file that is generated by the below code, and do a MongoRestore to a new collection on our Mongo servers.

      Please advice how I can get this working.

      Thanks

      marc

      private static bool ConvertObjects(FileStream sourceFile, FileStream destFileStream)
              {
                  ActiveRecommendations activeRec = null;
                  long totalRecords = 0;
                  var loadStart = DateTime.Now;
      
                  try
                  {
      
                      using (var writer = new BsonBinaryWriter(destFileStream))
                      {
                          using (var reader = new BsonBinaryReader(sourceFile))
                          {
                              while (!reader.IsAtEndOfFile())
                              {
                                  activeRec = GetActiveRecommendation(BsonSerializer.Deserialize<EanSet>(reader));
                                  BsonSerializer.Serialize<ActiveRecommendations>(writer, activeRec);
      
                                  totalRecords++;
      
                                  if ((totalRecords % _reportInterval) == 0)
                                  {
                                      RenScribe.LogInfo($"Total records converted so far is {totalRecords:N0}.");
                                  }
                              }
                          }
                      }
                  }
                  catch (Exception ex)
                  {
      
                      throw;
                  }
      
                  RenScribe.LogInfo($"Total records converted {totalRecords:N0} in {DateTime.Now - loadStart}.");
      
                  return true;
      
              }
      

        1. Override.cs
          4 kB
          Marc Simkin
        2. OverrideRec.cs
          4 kB
          Marc Simkin
        3. Rec.cs
          2 kB
          Marc Simkin

            Assignee:
            robert@mongodb.com Robert Stam
            Reporter:
            msimkin@bn.com Marc Simkin
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: