Uploaded image for project: 'MongoDB Database Tools'
  1. MongoDB Database Tools
  2. TOOLS-2744

mongorestore not scaling due to unnecessary incremental sleep time

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical - P2
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 100.2.1
    • Component/s: None
    • Labels:
      None
    • Branch:
      v4.7

      Description

      Mongo restore not Scaling to due to unnecessary incremental sleep. 

      If mongorestore is run with a high --numInsertionWorkersPerCollection  or --numParallelCollections. The goroutine fail to spawn (in a timely fashion) and put the system in a high value sleep.

      This is due to github.com/mongodb/mongo-tools/mongorestore/restore.go :442

      "time.Sleep(time.Duration * 10 * time.Millisecond)"

      This line is redundant or should at-least be statically timed like "time.Sleep( 10 * time.Millisecond)"

      Each goroutine waits an incremental amount of time. Eg. Wait time before spawning first goroutine 10 ms , second 20 ms , so on till 40000ms for the 4000th thread. Sum of this AP comes to 80000000ms which comes to 80k secs. 

      A sample command that reproduces this issue is:-

      Taking a small  dump.bson will also reproduce this issue. The process simply wont complete even after reaching 100%. This also prevents any user to spawn more than 1300 go-routines in a practical timeline (1 day).

      Eg ./mongorestore --host 127.0.0.1:27300 --noIndexRestore --writeConcern "{w: 0}" --numInsertionWorkersPerCollection 1000 -vvvv /var/mongo/dump.bson

        Attachments

          Activity

            People

            Assignee:
            evgeni.dobranov Evgeni Dobranov
            Reporter:
            rakshitbansal25@gmail.com Rakshit Bansal
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: