Problem Statement/Rationale

If the buffer has already reached a limit that is near the flush threshold and the next model is over 100 bytes, the current logic in BufferedBulkInserter creates a bulk write command with existing buffer + the additional large document. The resulting command can be larger than 48MB, causing the command to be rejected by the server.

Expected Results

Mongorestore should not fail.

Actual Results

Mongorestore crash with the error "unable to write wire message to network".

Additional Notes

When receiving a new document, the BufferedBulkInserter could look ahead and calculate the size of its buffer + new document WriteModel size. If the size is greater than 48MB, it should flush the previous buffer, clear the buffer, and then add the new document in the buffer.

The size of message overhead depends on the size of the overhead depends on length of the namespace (255 bytes at max) and number of documents in the buffer. Currently, we're only estimating 100 bytes for this overhead, which is not enough for some cases. We should increase it aggressively to 1MB.