Uploaded image for project: 'MongoDB Database Tools'
  1. MongoDB Database Tools
  2. TOOLS-2372

BufferedBulkInserter doesn't limit memory usage before flush

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Accepted
    • Priority: Major - P3
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: mongo-tools-common
    • Labels:

      Description

      mongo-tools-common BufferedBulkInserter (which also allows updates, replaces, and deletes) allows specifying a document count, but doesn't have a way to limit memory usage with large documents. We should provide such a constraint.

      Two ways to address this:

      Faster solution: add size limit to mongo-tools-common

      Better solution: mongomirror has a different BufferedBulkInserter that only does inserts and flushes via if the documents reach the maximum doc size (16MB). We can reduce code duplication by merging these two inserters in a way that preserves the features that both sides need – rather than a "bulk inserter", create a "bulk operation buffer". mongomirror could set the maximum size to 16MB and only use the insert functionality, but mongo tools could use a larger size if they so chose.

        Attachments

          Activity

            People

            Assignee:
            Unassigned
            Reporter:
            david.golden David Golden
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated: