Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-33406

Reduce concurrency for s390x compile and compile_all builds to avoid OOM failures

    • Type: Icon: Task Task
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 3.4.14, 3.6.4, 3.7.3
    • Affects Version/s: None
    • Component/s: Build
    • None
    • Fully Compatible
    • v3.6, v3.4
    • 0

      The compile and compile_all Evergreen tasks has been failing intermittently due to one of the ld processes being killed by OOM killer since Feb 2017. As a workaround—similar to what was done in SERVER-29442 and SERVER-30516—we should reduce the value for the --jobs option specified to SCons when our s390x builders are running the compile and compile_all task to avoid further Evergreen failures. Given that an individual linker process may use >3.5GB of memory of 16GB total, we should prevent SCons from running more than 3 of them as the same time by specifying -j3 rather than -j$(grep -c ^processor /proc/cpuinfo) or -j$(( $(grep -c ^processor /proc/cpuinfo) / 2 )) in each build variant's ${compile_flags} expansion.

      [2018/02/16 20:07:33.596] mci-exec  9539  3.3 22.5 3896412 3660772 ?     D    19:52   0:31 /opt/mongodbtoolchain/v2/lib/gcc/s390x-mongodb-linux/5.4.0/../../../../s390x-mongodb-linux/bin/ld @/tmp/ccoOHhDJ
      [2018/02/16 20:07:33.596] mci-exec  9837  3.4 22.6 3896592 3685484 ?     D    19:52   0:31 /opt/mongodbtoolchain/v2/lib/gcc/s390x-mongodb-linux/5.4.0/../../../../s390x-mongodb-linux/bin/ld @/tmp/cc8EneGh
      [2018/02/16 20:07:33.597] mci-exec 13693  5.0 22.2 3883392 3625496 ?     D    19:57   0:30 /opt/mongodbtoolchain/v2/lib/gcc/s390x-mongodb-linux/5.4.0/../../../../s390x-mongodb-linux/bin/ld @/tmp/ccLraHsk
      [2018/02/16 20:07:33.597] mci-exec 14395  5.5 22.3 3885152 3640884 ?     D    19:58   0:30 /opt/mongodbtoolchain/v2/lib/gcc/s390x-mongodb-linux/5.4.0/../../../../s390x-mongodb-linux/bin/ld @/tmp/ccekp0ZZ
      

      https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_ubuntu1604_s390x_compile_all_537e479caf3c6a2db67c0fa1c5d6769c9aea8786_01_01_01_00_00_00

            Assignee:
            ian@mongodb.com Ian Whalen (Inactive)
            Reporter:
            max.hirschhorn@mongodb.com Max Hirschhorn
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: