Uploaded image for project: 'MongoDB Database Tools'
  1. MongoDB Database Tools
  2. TOOLS-2372

BufferedBulkInserter doesn't limit memory usage before flush

    XMLWordPrintable

Details

    • Bug
    • Status: Accepted
    • Major - P3
    • Resolution: Unresolved
    • None
    • None
    • mongo-tools-common

    Description

      mongo-tools-common BufferedBulkInserter (which also allows updates, replaces, and deletes) allows specifying a document count, but doesn't have a way to limit memory usage with large documents. We should provide such a constraint.

      Two ways to address this:

      Faster solution: add size limit to mongo-tools-common

      Better solution: mongomirror has a different BufferedBulkInserter that only does inserts and flushes via if the documents reach the maximum doc size (16MB). We can reduce code duplication by merging these two inserters in a way that preserves the features that both sides need – rather than a "bulk inserter", create a "bulk operation buffer". mongomirror could set the maximum size to 16MB and only use the insert functionality, but mongo tools could use a larger size if they so chose.

      Attachments

        Activity

          People

            Unassigned Unassigned
            david.golden@mongodb.com David Golden
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: