There are a couple possible approaches to take here:
- add a user configurable knob (pros: easy and easily tunable; cons: easy to misuse)
- do an adaptive solution (pros: just works; cons: tricky to get right)
- It will be easy to overload the server
- Need to change how users throttle writes
- If it is a requirement that the import is sequential according to the source file, parallel import is not possible
Because mongoimport works against a single collection, it is the easiest tool to parallelize. Simply read-ahead from the source into memory and use >1 connection to the server.