Uploaded image for project: 'MongoDB Database Tools'
  1. MongoDB Database Tools
  2. TOOLS-1856

use a memory pool in mongorestore

XMLWordPrintableJSON

    • 2
    • 1,156

      Instead of allocating a new array for each document to restore (cf line 268), mongorestore should use a memory pool in order to reduce pressure on GC.

      I tried to implement it and got a ~25% speedup when restoring 2 collections with 10 millions documents:

      Unable to find source-code formatter for language: shell. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
      > du -h -a dump/test
4,0K	dump/test/link.metadata.json
3,0G	dump/test/test.bson
4,0K	dump/test/test.metadata.json
535M	dump/test/link.bson
3,5G	dump/test

      old version:

      Unable to find source-code formatter for language: shell. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
      > time mongorestore --dir dump
mongorestore --dir dump/  62,65s user 21,19s system 34% cpu 4:03,84 total

      new version:

      Unable to find source-code formatter for language: shell. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
      > time mongorestore --dir dump
mongorestore --dir dump  54,98s user 19,31s system 39% cpu 3:08,43 total

            Assignee:
            Unassigned Unassigned
            Reporter:
            felix2626 adrien petel
            Huan Li, Matthew Chiaravalloti, Ryan Chipman, Tim Fogarty
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: