Uploaded image for project: 'MongoDB Database Tools'
  1. MongoDB Database Tools
  2. TOOLS-3300

MongoDB tools should use maxBsonObjectSize (16MB) as the buffered bulk inserter byte limit

    • Type: Icon: Task Task
    • Resolution: Unresolved
    • Priority: Icon: Minor - P4 Minor - P4
    • None
    • Affects Version/s: None
    • Component/s: All Tools
    • None
    • 1
    • 549

      Problem

      The go-driver will split up bulk insert commands whose documents exceed 16MB, but all MongoDB tools incorrectly assume that bulk insert commands whose documents total size does not exceed 48MB will be inserted as a single command. This makes MongoDB tools unnecessarily wait to fill up at 48MB buffer before flushing the content to the database.

      Relevant thread: https://mongodb.slack.com/archives/C7WJZNUTA/p1682000864772909 

      Relevant documentation: https://www.mongodb.com/docs/manual/reference/command/hello/#mongodb-data-hello.maxBsonObjectSize 

      Solution & Acceptance Criteria

      Update MongoDB tools to use the correct 16MB (16 * 1024 * 1024) limit when inserting documents in bulk.

      Code location: https://github.com/mongodb/mongo-tools/blob/306328af6931516b4344e32c2823f9294472456c/common/db/buffered_bulk.go#L20

      Impact

      MongoDB tools will no longer unnecessarily wait to fill a 48MB buffer before flushing the content, making the tools use the server resources more effectively.

            Assignee:
            Unassigned Unassigned
            Reporter:
            craven.huynh@mongodb.com Craven Huynh (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: