Uploaded image for project: 'MongoDB Database Tools'
  1. MongoDB Database Tools
  2. TOOLS-1856

use a memory pool in mongorestore

    XMLWordPrintableJSON

Details

    • 2

    Description

      Instead of allocating a new array for each document to restore (cf line 268), mongorestore should use a memory pool in order to reduce pressure on GC.

      I tried to implement it and got a ~25% speedup when restoring 2 collections with 10 millions documents: 

      > du -h -a dump/test
      		 
      4,0K	dump/test/link.metadata.json
      		 
      3,0G	dump/test/test.bson
      		 
      4,0K	dump/test/test.metadata.json
      		 
      535M	dump/test/link.bson
      		 
      3,5G	dump/test

      old version:

      > time mongorestore --dir dump
      		 
      mongorestore --dir dump/  62,65s user 21,19s system 34% cpu 4:03,84 total

      new version: 

      > time mongorestore --dir dump
      		 
      mongorestore --dir dump  54,98s user 19,31s system 39% cpu 3:08,43 total

      Attachments

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. TOOLS-2783 Mongorestore uses huge amount of RAM

          • Major - P3 - Major loss of function.
          • Closed
          is depended on by

          Task - A task that needs to be done. TOOLS-2665 Optimize usage of Result type in Mongorestore

          • Major - P3 - Major loss of function.
          • Waiting (Blocked)
          is related to

          Task - A task that needs to be done. TOOLS-2665 Optimize usage of Result type in Mongorestore

          • Major - P3 - Major loss of function.
          • Waiting (Blocked)
          related to

          Task - A task that needs to be done. TOOLS-2875 Limit the BufferedBulkInserter's batch size by bytes

          • Major - P3 - Major loss of function.
          • Closed

          Investigation - TOOLS-2642 Investigate and implement performance-related optimizations in mongoimport

          • Major - P3 - Major loss of function.
          • Accepted
          links to

          GitHub Pull Request (mongodb/mongo-tools master)

          Activity

            People

              Unassigned Unassigned
              felix2626 adrien petel
              Huan Li, Matthew Chiaravalloti, Ryan Chipman, Tim Fogarty
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: