-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: 8.0.8
-
Component/s: None
-
Workload Resilience
-
ALL
-
-
Workload Scheduling 2025-09-15, Workload Resilience 2025-09-29, Workload Resilience 2025-10-13, Workload Resilience 2025-10-27, Workload Resilience 2025-11-10
-
None
-
None
-
None
-
None
-
None
-
None
-
None
We were trying to figure out why the per-CPU memory pools for tcmalloc were not enabled in mongodb v8. Everything appeared to be set correctly.
We would see that `usingPerCPUCaches` is `true` but `cpu_free` was zero.
rs0 [primary] formative> db.runCommand( \{ serverStatus: 1, tcmalloc: 2 } ).tcmalloc.tcmalloc.cpu_free
0
rs0 [primary] formative> db.runCommand( \{ serverStatus: 1, tcmalloc: 2 } ).tcmalloc.usingPerCPUCaches
true
rs0 [primary] formative> db.serverBuildInfo()
{
{ version: '8.0.8-3',}
{ psmdbVersion: '8.0.8-3',}
{ gitVersion: '6c9bddbcdcbc766a0771e4756e1310252b2c91de',}
{allocator: 'tcmalloc-google',}
{ javascriptEngine: 'mozjs',}
{openssl: {}
{ running: 'OpenSSL 3.2.2 4 Jun 2024',}
{{ compiled: 'OpenSSL 3.2.2 4 Jun 2024'}}\
{,}}
}
I found that setting the environment variable `MONGO_TCMALLOC_PER_CPU_CACHE_SIZE_BYTES=10485760` resolves the issue.
I noticed that mongod uses /sys/fs/cgroup/memory.max to calculate max memory for calculating the max memory per CPU.
When I look at this file in the container it contains the string "max" instead of a number.
bash-5.1$ cat /sys/fs/cgroup/memory.max
max
I think this is because there's no limit set on the container.
Relevant code:
unsigned long long getMemorySizeLimitInBytes() { const unsigned long long systemMemBytes = getSystemMemorySizeBytes(); for (const char* file : { "/sys/fs/cgroup/memory.max", // cgroups v2 "/sys/fs/cgroup/memory/memory.limit_in_bytes" // cgroups v1 }) { unsigned long long groupMemBytes = 0; std::string groupLimit = parseLineFromFile(file); // Bug: should also ignore the string "max" ? Or maybe any string that doesn't parse as an integer? if (!groupLimit.empty() ) { return std::min(systemMemBytes, (unsigned long long)atoll(groupLimit.c_str())); } } return systemMemBytes; } }
Also relevant:
unsigned long long getMongoMaxCpuCacheSize(size_t numCpus) { char* userCacheSizeBytes = getenv("MONGO_TCMALLOC_PER_CPU_CACHE_SIZE_BYTES"); if (userCacheSizeBytes != nullptr) { auto value = atoll(userCacheSizeBytes); if (value != 0) { return value; } } // 1024MB in bytes spread across cores. size_t systemMemorySizeMB = getMemorySizeLimitInBytes() / (1024 * 1024); size_t defaultTcMallocPerCPUCacheSize = (1024 * 1024 * 1024) / numCpus; size_t derivedTcMallocPerCPUCacheSize = ((systemMemorySizeMB / 4) * 1024 * 1024) / numCpus; // 1/4 of system memory in bytes size_t perCPUCacheSize = std::min(defaultTcMallocPerCPUCacheSize, derivedTcMallocPerCPUCacheSize); return perCPUCacheSize; }