Uploaded image for project: 'MongoDB Database Tools'
  1. MongoDB Database Tools
  2. TOOLS-245

Add multithreaded support to mongoimport

    XMLWordPrintableJSON

Details

    • Task
    • Status: Closed
    • Major - P3
    • Resolution: Fixed
    • None
    • 2.8.0-rc0
    • mongoimport
    • None
    • MCI 2.7.8

    Description

      There are a couple possible approaches to take here:

      1. add a user configurable knob (pros: easy and easily tunable; cons: easy to misuse)
      2. do an adaptive solution (pros: just works; cons: tricky to get right)

      Other considerations:

      1. It will be easy to overload the server
      2. Need to change how users throttle writes
      3. If it is a requirement that the import is sequential according to the source file, parallel import is not possible

      Because mongoimport works against a single collection, it is the easiest tool to parallelize. Simply read-ahead from the source into memory and use >1 connection to the server.

      Attachments

        Issue Links

          Activity

            People

              adinoyi.omuya@mongodb.com Adinoyi Omuya
              ian@mongodb.com Ian Whalen (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: