Uploaded image for project: 'MongoDB Database Tools'
  1. MongoDB Database Tools
  2. TOOLS-245

Add multithreaded support to mongoimport

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0-rc0
    • Component/s: mongoimport
    • Labels:
      None
    • Sprint:
      MCI 2.7.8

      Description

      There are a couple possible approaches to take here:

      1. add a user configurable knob (pros: easy and easily tunable; cons: easy to misuse)
      2. do an adaptive solution (pros: just works; cons: tricky to get right)

      Other considerations:

      1. It will be easy to overload the server
      2. Need to change how users throttle writes
      3. If it is a requirement that the import is sequential according to the source file, parallel import is not possible

      Because mongoimport works against a single collection, it is the easiest tool to parallelize. Simply read-ahead from the source into memory and use >1 connection to the server.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              adinoyi.omuya Adinoyi Omuya
              Reporter:
              ian.whalen Ian Whalen
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: