We were trying to figure out why the per-CPU memory pools for tcmalloc were not enabled in mongodb v8.  Everything appeared to be set correctly.

We would see that `usingPerCPUCaches` is `true` but `cpu_free` was zero.

rs0 [primary] formative> db.runCommand( { serverStatus: 1, tcmalloc: 2 } ).tcmalloc.tcmalloc.cpu_free
0
rs0 [primary] formative> db.runCommand( { serverStatus: 1, tcmalloc: 2 } ).tcmalloc.usingPerCPUCaches
true
rs0 [primary] formative> db.serverBuildInfo()
{{{}}
  version: '8.0.8-3',
  psmdbVersion: '8.0.8-3',
  gitVersion: '6c9bddbcdcbc766a0771e4756e1310252b2c91de',
allocator: 'tcmalloc-google',
  javascriptEngine: 'mozjs',
openssl: {
    running: 'OpenSSL 3.2.2 4 Jun 2024',
    compiled: 'OpenSSL 3.2.2 4 Jun 2024'
  },
}

I found that setting the environment variable `MONGO_TCMALLOC_PER_CPU_CACHE_SIZE_BYTES=10485760` resolves the issue.

I noticed that mongod uses /sys/fs/cgroup/memory.max to calculate max memory for calculating the max memory per CPU.

When I look at this file in the container it contains the string "max" instead of a number.

bash-5.1$ cat /sys/fs/cgroup/memory.max
max

I think this is because there's no limit set on the container.

Relevant code:

unsigned long long getMemorySizeLimitInBytes() {
    const unsigned long long systemMemBytes = getSystemMemorySizeBytes();
    for (const char* file : {
              "/sys/fs/cgroup/memory.max",                   // cgroups v2
              "/sys/fs/cgroup/memory/memory.limit_in_bytes"  // cgroups v1
          }) {

        unsigned long long groupMemBytes = 0;
        std::string groupLimit = parseLineFromFile(file);
// Bug: should also ignore the string "max" ?  Or maybe any string that doesn't parse as an integer?
        if (!groupLimit.empty() ) { 
            return std::min(systemMemBytes, (unsigned long long)atoll(groupLimit.c_str()));
{{        }}}
{{    }}}
    return systemMemBytes;
}

}

Also relevant:

unsigned long long getMongoMaxCpuCacheSize(size_t numCpus) {

    char* userCacheSizeBytes = getenv("MONGO_TCMALLOC_PER_CPU_CACHE_SIZE_BYTES");
    if (userCacheSizeBytes != nullptr) {
        auto value = atoll(userCacheSizeBytes);
        if (value != 0) {
            return value;
{{        }}}
{{    }}}

    // 1024MB in bytes spread across cores.
    size_t systemMemorySizeMB = getMemorySizeLimitInBytes() / (1024 * 1024);
    size_t defaultTcMallocPerCPUCacheSize = (1024 * 1024 * 1024) / numCpus;
    size_t derivedTcMallocPerCPUCacheSize =
        ((systemMemorySizeMB / 4) * 1024 * 1024) / numCpus;  // 1/4 of system memory in bytes

    size_t perCPUCacheSize =
        std::min(defaultTcMallocPerCPUCacheSize, derivedTcMallocPerCPUCacheSize);
    return perCPUCacheSize;
}