Uploaded image for project: 'MongoDB Database Tools'
  1. MongoDB Database Tools
  2. TOOLS-245

Add multithreaded support to mongoimport

    • Type: Icon: Task Task
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 2.8.0-rc0
    • Affects Version/s: None
    • Component/s: mongoimport
    • Labels:
    • MCI 2.7.8

      There are a couple possible approaches to take here:

      1. add a user configurable knob (pros: easy and easily tunable; cons: easy to misuse)
      2. do an adaptive solution (pros: just works; cons: tricky to get right)

      Other considerations:

      1. It will be easy to overload the server
      2. Need to change how users throttle writes
      3. If it is a requirement that the import is sequential according to the source file, parallel import is not possible

      Because mongoimport works against a single collection, it is the easiest tool to parallelize. Simply read-ahead from the source into memory and use >1 connection to the server.

            adinoyi.omuya@mongodb.com Adinoyi Omuya
            ian@mongodb.com Ian Whalen (Inactive)
            0 Vote for this issue
            2 Start watching this issue