Uploaded image for project: 'MongoDB Database Tools'
  1. MongoDB Database Tools
  2. TOOLS-2744

mongorestore not scaling due to unnecessary incremental sleep time

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Critical - P2 Critical - P2
    • 100.2.1
    • Affects Version/s: None
    • Component/s: None
    • None
    • v4.7

      Mongo restore not Scaling to due to unnecessary incremental sleep. 

      If mongorestore is run with a high --numInsertionWorkersPerCollection  or --numParallelCollections. The goroutine fail to spawn (in a timely fashion) and put the system in a high value sleep.

      This is due to github.com/mongodb/mongo-tools/mongorestore/restore.go :442

      "time.Sleep(time.Duration * 10 * time.Millisecond)"

      This line is redundant or should at-least be statically timed like "time.Sleep( 10 * time.Millisecond)"

      Each goroutine waits an incremental amount of time. Eg. Wait time before spawning first goroutine 10 ms , second 20 ms , so on till 40000ms for the 4000th thread. Sum of this AP comes to 80000000ms which comes to 80k secs. 

      A sample command that reproduces this issue is:-

      Taking a small  dump.bson will also reproduce this issue. The process simply wont complete even after reaching 100%. This also prevents any user to spawn more than 1300 go-routines in a practical timeline (1 day).

      Eg ./mongorestore --host --noIndexRestore --writeConcern "{w: 0}" --numInsertionWorkersPerCollection 1000 -vvvv /var/mongo/dump.bson

            evgeni.dobranov@mongodb.com Evgeni Dobranov
            rakshitbansal25@gmail.com Rakshit Bansal
            0 Vote for this issue
            6 Start watching this issue