Evergreen builds can fail to allocate native memory

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Fixed
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Build
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Occasionally Evergreen builds fail with the following error:

      [2020/04/22 14:34:01.679] # There is insufficient memory for the Java Runtime Environment to continue.
      [2020/04/22 14:34:01.679] # Native memory allocation (mmap) failed to map 133693440 bytes for committing reserved memory.
      [2020/04/22 14:34:01.679] # An error report file with more information is saved as:
      [2020/04/22 14:34:01.679] # /data/mci/1631215de5e973483dc5894d8e123c9c/src/driver-reactive-streams/hs_err_pid5312.log
      [2020/04/22 14:34:01.786] Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000078d080000, 133693440, 0) failed; error='Cannot allocate memory' (errno=12)
      

      For example, see https://evergreen.mongodb.com/task/mongo_java_driver_tests_zlib_compression__version~latest_os~linux_topology~standalone_auth~noauth_ssl~nossl_compressor~zlib_jdk~jdk8_test_3327634ef154a55875e9b24d69f5577c5e3e3531_20_04_09_21_30_18

      It seems correlated with variants that enable zlib compression.

      Googling around a bit the only thing potentially useful I found was https://bugs.openjdk.java.net/browse/JDK-8187709, which links to https://blogs.oracle.com/poonam/running-on-a-64bit-platform-and-still-running-out-of-memory.

      Based on those source there are a couple of things we can try:

      • Run with -XX:-UseCompressedOops
      • Keep the CompressedOops and set the base of the Java Heap with the JVM option -XX:HeapBaseMinAddress=n to specify the address where the Java Heap should start from

      Based on the hot spot log file (attached), it does seem like we are low on memory, so we can try using a server with more physical memory available.

            Assignee:
            Jeffrey Yemin
            Reporter:
            Jeffrey Yemin
            None
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: