Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-26224

Resmoke gets much slower with more parallelism

    • Fully Compatible
    • ALL
    • Hide
      % resmoke -j24 --suite=decimal
      

      yields: All 28 test(s) passed in 200.13 seconds,
      but

      % resmoke -j1 --suite=decimal
      

      yields: All 28 test(s) passed in 22.45 seconds.

      Show
      % resmoke -j24 --suite=decimal yields: All 28 test(s) passed in 200.13 seconds , but % resmoke -j1 --suite=decimal yields: All 28 test(s) passed in 22.45 seconds .
    • TIG 2017-04-17

      With -j24 test suites now take at a minimum 200 seconds to execute, while some used to run in about 5 seconds. This slows down development and probably adds to CI costs.

      The following fixes it:

      diff --git a/buildscripts/resmokelib/testing/executor.py b/buildscripts/resmokelib/testing/executor.py
      index 3628fa0..0ce5ef9 100644
      --- a/buildscripts/resmokelib/testing/executor.py
      +++ b/buildscripts/resmokelib/testing/executor.py
      @@ -149,10 +149,6 @@ class TestGroupExecutor(object):
                       t.daemon = True
                       t.start()
                       threads.append(t)
      -                # SERVER-24729 Need to stagger when jobs start to reduce I/O load if there
      -                # are many of them.  Both the 5 and the 10 are arbitrary.
      -                if len(threads) >= 5:
      -                    time.sleep(10)
       
                   joined = False
                   while not joined:
      

      With this,

      % resmoke -j1 --suite=decimal
      

      yields: All 28 test(s) passed in 5.41 seconds.
      This is a 37 times speedup.

            Assignee:
            eddie.louie Eddie Louie
            Reporter:
            geert.bosch@mongodb.com Geert Bosch
            Votes:
            1 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: