BufferedBulkInserter should keep batch size under 48MB

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Duplicate
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: mongorestore
    • 3
    • Tools and Replicator

      Problem Statement/Rationale

      If the buffer has already reached a limit that is near the flush threshold and the next model is over 100 bytes, the current logic in BufferedBulkInserter creates a bulk write command with existing buffer + the additional large document. The resulting command can be larger than 48MB, causing the command to be rejected by the server.

      Expected Results

      Mongorestore should not fail.

      Actual Results

      Mongorestore crash with the error "unable to write wire message to network".

      Additional Notes

      When receiving a new document, the BufferedBulkInserter could look ahead and calculate the size of its buffer + new document WriteModel size. If the size is greater than 48MB, it should flush the previous buffer, clear the buffer, and then add the new document in the buffer.

            Assignee:
            Jian Guan
            Reporter:
            Jian Guan
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: