Uploaded image for project: 'MongoDB Database Tools'
  1. MongoDB Database Tools
  2. TOOLS-2372

BufferedBulkInserter doesn't limit memory usage before flush

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: mongo-tools-common
    • Labels:

      mongo-tools-common BufferedBulkInserter (which also allows updates, replaces, and deletes) allows specifying a document count, but doesn't have a way to limit memory usage with large documents. We should provide such a constraint.

      Two ways to address this:

      Faster solution: add size limit to mongo-tools-common

      Better solution: mongomirror has a different BufferedBulkInserter that only does inserts and flushes via if the documents reach the maximum doc size (16MB). We can reduce code duplication by merging these two inserters in a way that preserves the features that both sides need – rather than a "bulk inserter", create a "bulk operation buffer". mongomirror could set the maximum size to 16MB and only use the insert functionality, but mongo tools could use a larger size if they so chose.

            Unassigned Unassigned
            david.golden@mongodb.com David Golden
            0 Vote for this issue
            1 Start watching this issue