Uploaded image for project: 'MongoDB Database Tools'
  1. MongoDB Database Tools
  2. TOOLS-2875

Limit the BufferedBulkInserter's batch size by bytes

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Accepted
    • Priority: Major - P3
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 100.6.0
    • Component/s: None
    • Labels:
      None
    • Case:

      Description

      As part of TOOLS-1956 we removed the byte limit on batch sizes in the BufferedBulkInserter. (See mtc and tools.)

      This means each batch of the BufferedBulkInserter can hold up to ~16 GB of data before it gets flushed. The theoretical maximum of data that can be stored in BufferedBulkInserter's in mongorestore is ~16 GB * NumParallelCollections * NumInsertionWorkers. This is ~64 GB by default.

      This can have a severe impact on performance, even for average document sizes of 1-2 MB.

      We should add a new --batchMB option to mongorestore and mongoimport. The BufferedBulkInserter will flush its batch whenever the document count reaches the batchSize OR the total size of documents in the batch reaches batchMB.

      I think 16 is a reasonable default for batchMB. I did not see performance increases for larger values of batchMB in my tests, though that's probably system and workload dependent.

      This will provide a sane default for the memory usage of BufferedBulkInserter, while allowing users to increase the value of they think it will improve performance for their use case. Alternatively, we could set the limit but not make it tunable by a command line option.

      Additionally, this will provide a limit for the size of the sync.Pool in TOOLS-1856.

       

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              tim.fogarty Tim Fogarty
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated: