Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-33406

Reduce concurrency for s390x compile and compile_all builds to avoid OOM failures

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.4.14, 3.6.4, 3.7.3
    • Component/s: Build
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Backport Requested:
      v3.6, v3.4
    • Linked BF Score:
      0

      Description

      The compile and compile_all Evergreen tasks has been failing intermittently due to one of the ld processes being killed by OOM killer since Feb 2017. As a workaround—similar to what was done in SERVER-29442 and SERVER-30516—we should reduce the value for the --jobs option specified to SCons when our s390x builders are running the compile and compile_all task to avoid further Evergreen failures. Given that an individual linker process may use >3.5GB of memory of 16GB total, we should prevent SCons from running more than 3 of them as the same time by specifying -j3 rather than -j$(grep -c ^processor /proc/cpuinfo) or -j$(( $(grep -c ^processor /proc/cpuinfo) / 2 )) in each build variant's ${compile_flags} expansion.

      [2018/02/16 20:07:33.596] mci-exec  9539  3.3 22.5 3896412 3660772 ?     D    19:52   0:31 /opt/mongodbtoolchain/v2/lib/gcc/s390x-mongodb-linux/5.4.0/../../../../s390x-mongodb-linux/bin/ld @/tmp/ccoOHhDJ
      [2018/02/16 20:07:33.596] mci-exec  9837  3.4 22.6 3896592 3685484 ?     D    19:52   0:31 /opt/mongodbtoolchain/v2/lib/gcc/s390x-mongodb-linux/5.4.0/../../../../s390x-mongodb-linux/bin/ld @/tmp/cc8EneGh
      [2018/02/16 20:07:33.597] mci-exec 13693  5.0 22.2 3883392 3625496 ?     D    19:57   0:30 /opt/mongodbtoolchain/v2/lib/gcc/s390x-mongodb-linux/5.4.0/../../../../s390x-mongodb-linux/bin/ld @/tmp/ccLraHsk
      [2018/02/16 20:07:33.597] mci-exec 14395  5.5 22.3 3885152 3640884 ?     D    19:58   0:30 /opt/mongodbtoolchain/v2/lib/gcc/s390x-mongodb-linux/5.4.0/../../../../s390x-mongodb-linux/bin/ld @/tmp/ccekp0ZZ
      

      https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_ubuntu1604_s390x_compile_all_537e479caf3c6a2db67c0fa1c5d6769c9aea8786_01_01_01_00_00_00

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: