Add multithreaded support to mongoimport

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Done
    • Priority: Major - P3
    • 2.8.0-rc0
    • Affects Version/s: None
    • Component/s: mongoimport
    • None
    • MCI 2.7.8

      There are a couple possible approaches to take here:

      1. add a user configurable knob (pros: easy and easily tunable; cons: easy to misuse)
      2. do an adaptive solution (pros: just works; cons: tricky to get right)

      Other considerations:

      1. It will be easy to overload the server
      2. Need to change how users throttle writes
      3. If it is a requirement that the import is sequential according to the source file, parallel import is not possible

      Because mongoimport works against a single collection, it is the easiest tool to parallelize. Simply read-ahead from the source into memory and use >1 connection to the server.

              Assignee:
              Adinoyi Omuya (Inactive)
              Reporter:
              Ian Whalen (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: