[JAVA-3709] Evergreen builds can fail to allocate native memory Created: 22/Apr/20  Updated: 28/Oct/23  Resolved: 23/Apr/20

Status: Closed
Project: Java Driver
Component/s: Build
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Jeffrey Yemin Assignee: Jeffrey Yemin
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File mongo_java_driver_tests_zlib_compression__version_latest_os_linux_topology_standalone_auth_noauth_ssl_nossl_compressor_zlib_jdk_jdk8_test_3327634ef154a55875e9b24d69f5577c5e3e3531_20_04_09_21_30_18-0-working-dir-logs.tar.gz    

 Description   

Occasionally Evergreen builds fail with the following error:

[2020/04/22 14:34:01.679] # There is insufficient memory for the Java Runtime Environment to continue.
[2020/04/22 14:34:01.679] # Native memory allocation (mmap) failed to map 133693440 bytes for committing reserved memory.
[2020/04/22 14:34:01.679] # An error report file with more information is saved as:
[2020/04/22 14:34:01.679] # /data/mci/1631215de5e973483dc5894d8e123c9c/src/driver-reactive-streams/hs_err_pid5312.log
[2020/04/22 14:34:01.786] Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000078d080000, 133693440, 0) failed; error='Cannot allocate memory' (errno=12)

For example, see https://evergreen.mongodb.com/task/mongo_java_driver_tests_zlib_compression__version~latest_os~linux_topology~standalone_auth~noauth_ssl~nossl_compressor~zlib_jdk~jdk8_test_3327634ef154a55875e9b24d69f5577c5e3e3531_20_04_09_21_30_18

It seems correlated with variants that enable zlib compression.

Googling around a bit the only thing potentially useful I found was https://bugs.openjdk.java.net/browse/JDK-8187709, which links to https://blogs.oracle.com/poonam/running-on-a-64bit-platform-and-still-running-out-of-memory.

Based on those source there are a couple of things we can try:

  • Run with -XX:-UseCompressedOops
  • Keep the CompressedOops and set the base of the Java Heap with the JVM option -XX:HeapBaseMinAddress=n to specify the address where the Java Heap should start from

Based on the hot spot log file (attached), it does seem like we are low on memory, so we can try using a server with more physical memory available.



 Comments   
Comment by Githook User [ 24/Apr/20 ]

Author:

{'name': 'Jeff Yemin', 'email': 'jeff.yemin@mongodb.com', 'username': 'jyemin'}

Message: Use larger servers to run zlib tests in order to avoid memory exhaustion

JAVA-3709
Branch: 4.0.x
https://github.com/mongodb/mongo-java-driver/commit/1a7e2dc861ef7c43f4b9e058d356789115dea276

Comment by Githook User [ 23/Apr/20 ]

Author:

{'name': 'Jeff Yemin', 'email': 'jeff.yemin@mongodb.com', 'username': 'jyemin'}

Message: Use larger servers to run zlib tests in order to avoid memory exhaustion

JAVA-3709
Branch: master
https://github.com/mongodb/mongo-java-driver/commit/575bc8c308ec4db9eb0b5be940841a259b1ff2be

Comment by Jeffrey Yemin [ 22/Apr/20 ]

From the hot spot log file (attached):

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 49283072 bytes for committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (os_linux.cpp:2640), pid=5557, tid=0x00007f973621e700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_162-b12) (build 1.8.0_162-b12)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.162-b12 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
 
---------------  T H R E A D  ---------------
 
Current thread (0x00007f9750078000):  VMThread [stack: 0x00007f973611e000,0x00007f973621f000] [id=5564]
 
Stack: [0x00007f973611e000,0x00007f973621f000],  sp=0x00007f973621d260,  free space=1020k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xacfb2a]  VMError::report_and_die()+0x2ba
V  [libjvm.so+0x50060b]  report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0x8b
V  [libjvm.so+0x92d963]  os::Linux::commit_memory_impl(char*, unsigned long, bool)+0x123
V  [libjvm.so+0x92db89]  os::pd_commit_memory(char*, unsigned long, unsigned long, bool)+0x29
V  [libjvm.so+0x924e4a]  os::commit_memory(char*, unsigned long, unsigned long, bool)+0x2a
V  [libjvm.so+0x99adf3]  PSVirtualSpace::expand_by(unsigned long)+0x53
V  [libjvm.so+0x99c1f8]  PSYoungGen::resize_generation(unsigned long, unsigned long)+0xf8
V  [libjvm.so+0x99b352]  PSYoungGen::resize(unsigned long, unsigned long)+0x22
V  [libjvm.so+0x99862b]  PSScavenge::invoke_no_policy()+0xf3b
V  [libjvm.so+0x998db1]  PSScavenge::invoke()+0x41
V  [libjvm.so+0x94f880]  ParallelScavengeHeap::failed_mem_allocate(unsigned long)+0x70
V  [libjvm.so+0xad15a7]  VM_ParallelGCFailedAllocation::doit()+0x97
V  [libjvm.so+0xad90c5]  VM_Operation::evaluate()+0x55
V  [libjvm.so+0xad748a]  VMThread::evaluate_operation(VM_Operation*)+0xba
V  [libjvm.so+0xad780e]  VMThread::loop()+0x1ce
V  [libjvm.so+0xad7c80]  VMThread::run()+0x70
V  [libjvm.so+0x92dcc8]  java_start(Thread*)+0x108
 
VM_Operation (0x00007f96ade9d450): ParallelGCFailedAllocation, mode: safepoint, requested by thread 0x00007f96fb78f000
 
...
 
---------------  S Y S T E M  ---------------
 
OS:Red Hat Enterprise Linux Server release 7.0 (Maipo)
 
uname:Linux 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 9 10:09:10 EDT 2016 x86_64
libc:glibc 2.17 NPTL 2.17
rlimit: STACK 8192k, CORE 0k, NPROC 64000, NOFILE 64000, AS infinity
load average:1.80 1.49 0.73
 
/proc/meminfo:
MemTotal:        7231292 kB
MemFree:          125324 kB
MemAvailable:      60380 kB
Buffers:             352 kB

Generated at Thu Feb 08 09:00:14 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.