Uploaded image for project: 'MongoDB Database Tools'
  1. MongoDB Database Tools
  2. TOOLS-1856

use a memory pool in mongorestore

    XMLWordPrintable

    Details

      Description

      Instead of allocating a new array for each document to restore (cf line 268), mongorestore should use a memory pool in order to reduce pressure on GC.

      I tried to implement it and got a ~25% speedup when restoring 2 collections with 10 millions documents:

      > du -h -a dump/test
      4,0K	dump/test/link.metadata.json
      3,0G	dump/test/test.bson
      4,0K	dump/test/test.metadata.json
      535M	dump/test/link.bson
      3,5G	dump/test
      

      old version:

      > time mongorestore --dir dump
      mongorestore --dir dump/  62,65s user 21,19s system 34% cpu 4:03,84 total
      

      new version:

      > time mongorestore --dir dump
      mongorestore --dir dump  54,98s user 19,31s system 39% cpu 3:08,43 total
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              tim.fogarty Tim Fogarty
              Reporter:
              felix2626 adrien petel
              Reviewers:
              Huan Li, Matthew Chiaravalloti, Ryan Chipman, Tim Fogarty
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: