Uploaded image for project: 'MongoDB Command Line Tools'
  1. MongoDB Command Line Tools
  2. TOOLS-119

make mongorestore multi-threaded / parallel

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor - P4
    • Resolution: Fixed
    • Affects Version/s: Legacy C++ Implementation
    • Fix Version/s: 2.8.0-rc0
    • Component/s: mongorestore
    • Labels:
    • Documentation Changes:
      Completed
    • Sprint:
      MCI 2.7.8

      Description

      There are a couple different ways to make this parallelized:

      1. On the server, use the multi-index builder to build them in parallel. As of 2.6, the createIndexes command can take >1 index at a time. As of 2.8, it also creates them in parallel, so using a single createIndexes for all indexes for a collection should make index builds much more efficient.
      2. On the client:
        1. Open one connection per collection (up to a user-specified or adaptively arrived at limit)
        2. Use >1 thread per collection. Read ahead from the source file and make multiple connections to the server

      Considerations:

      1. Throttling the rate from the client to server so as not to overload either client or server.
      2. When using >1 thread per collection, data will not be restored in the same order it was dumped.

      Original Description
      I am running mongorestore to recreate a copy of large(ish) production database on a separate system (~300GB). It seems from observation that the process of importing the data and re-creating the indexes is happening in serial. Given that indexes can be created in the background during normal operating conditions, that at least this bit could be done in parallel. Ideally it would be fantastic to see the collections themselves be restored in parallel since the machine(s) I'm working with have plenty of extra resources to spare for this process. Is this doable? Or perhaps there are complexities that prevent this which I am not aware of?
      Thanks, as always.

        Attachments

          Issue Links

          There are no Sub-Tasks for this issue.

            Activity

              People

              • Assignee:
                kyle.erf Kyle Erf
                Reporter:
                scott.d'aquila@morganstanley.com Scott D'Aquila
              • Votes:
                8 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: