• Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: 7.0.16
    • Component/s: None
    • Environment:
      - `MongoDB 7.0.16, FCV 7.0, sharded cluster (32 shards), container with cgroup v2 limit 32 GiB on a 1 TB host, no lxcfs.`
    • ALL
    • Hide
      1. Run mongod in a container with cgroup v2 memory.max = 32 GiB, on a host with 1 TB RAM, without lxcfs mounted.
      2.  
        cat /proc/meminfo | head -1 
        ->   MemTotal: 1055766528 kB   (the container sees the host value)
        
      1. Create a time-series collection and write continuously across many series.
      2. db.adminCommand({getParameter: 1, timeseriesIdleBucketExpiryMemoryUsageThreshold: 1})
              
        -> NumberLong("26394163200")i.e. 25/1024 of the host's 1,031,022 MB, not of the 32 GiB the process may use.
      1. db.serverStatus().bucketCatalog
              -> memoryUsage keeps growing while numBucketsClosedDueToMemoryThreshold stays 0.
      Show
      Run mongod in a container with cgroup v2 memory.max = 32 GiB, on a host with 1 TB RAM, without lxcfs mounted.   cat /proc/meminfo | head -1 ->   MemTotal: 1055766528 kB   (the container sees the host value) Create a time-series collection and write continuously across many series. db.adminCommand({getParameter: 1, timeseriesIdleBucketExpiryMemoryUsageThreshold: 1})       -> NumberLong( "26394163200" )i.e. 25/1024 of the host's 1,031,022 MB, not of the 32 GiB the process may use. db.serverStatus().bucketCatalog       -> memoryUsage keeps growing while numBucketsClosedDueToMemoryThreshold stays 0.
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Defect:

      The default value of timeseriesIdleBucketExpiryMemoryUsageThreshold is computed from host total memory via ProcessInfo::getSystemMemSizeMB(). In a container that does not mount lxcfs, /proc/meminfo reports the host's MemTotal, so the threshold can be far larger than the memory the process is actually allowed to use. Idle bucket expiry then never fires, and the bucket catalog grows until the container is OOM-killed.

      This is a per-call-site issue, not a defect in getSystemMemSizeMB(): that function's contract (host total memory) is correct, and hostInfo deliberately exposes both memSize and memLimit (SERVER-39966). The problem is that an allocation/reclamation threshold is derived from total memory instead of the effective limit – the class of bug described in SERVER-16571.

      Code locations:

      7.0 – src/mongo/db/timeseries/timeseries_global_options.cpp L36-L52:

       

      value = ProcessInfo::getSystemMemSizeMB() * 25 * 1024;
      

      The cgroup-aware counterpart already exists and is used for WiredTiger cache sizing: ProcessInfo::getMemSizeMB() returns memLimit, where memLimit = min(systemMemory, cgroupLimit) (processinfo_linux.cpp L621-L640, wiredtiger_util.cpp L586-L607).

       

      Observed values

      On shard s3 (primary), after several hours of continuous writes:

       

       

      PRIMARY> db.adminCommand({getParameter: 1, timeseriesIdleBucketExpiryMemoryUsageThreshold: 1})
      { "timeseriesIdleBucketExpiryMemoryUsageThreshold" : NumberLong("26394163200"), "ok" : 1 }
      
      PRIMARY> db.serverStatus().bucketCatalog
      {
        "numBuckets" : 2618944,
        "numOpenBuckets" : 2617510,
        "numIdleBuckets" : 2617510,
        "memoryUsage" : NumberLong("4432206461"),
        "numBucketsClosedDueToMemoryThreshold" : 0,
        "numBucketsArchivedDueToMemoryThreshold" : 0,
        "numBucketsClosedDueToCachePressure" : 2007289,
        ...
      }
      

      The automatic threshold is 26,394,163,200 B, which is 25/1024 of 1,031,022 MB of host memory. This host runs in a container with a 32 GiB cgroup limit, so a threshold derived from the effective limit would be 838,860,800 B (800 MiB). The current value is 31.5x larger.

      At the same moment:

      • memoryUsage is 4,432,206,461 B, i.e. 5.3x the threshold a correct derivation would produce;
      • numOpenBuckets equals numIdleBuckets (2,617,510), so the entire open set is idle and eligible for expiry; 
      • numBucketsClosedDueToMemoryThreshold and numBucketsArchivedDueToMemoryThreshold are both 0, so the memory-based expiry path has never run once.

      The catalog has grown more than 5x past the limit a correct threshold would impose, every open bucket is idle, and not a single bucket has been reclaimed. This is not a leak; the threshold is unreachable.

       

      Why this is a bug and not by design

      • SERVER-16571 established that allocation decisions must use the actual memory constraint, "for example when running MongoDB within containers". That fix covered WiredTiger cache, oplog sizing and tcmalloc; this threshold, introduced later in SERVER-61195 (5.0), was never added to that list.
      • SERVER-95511 states the goal as "a percentage that scales with the cluster", yet the implementation still reads host memory, so the percentage does not follow the container's size. On a 1 TB host with a 32 GiB limit it barely changes when the pod is resized.
      • The parameter is undocumented (see SERVER-95510), so "set it explicitly" is not a reasonable answer for the default.

       

      Impact

      The container limit is 32 GiB and the WiredTiger cache already claims 15.5 GiB of it, while the catalog's automatic threshold is 24.6 GiB (26,394,163,200 B) – together well past the limit before connections, indexes and query memory.

      Since the memory-based expiry path has never run (numBucketsClosedDueToMemoryThreshold = 0), growth is unbounded until the container is OOM-killed. This applies to any pod whose limit is a small fraction of host memory.

       

      Suggested fix

      Derive the automatic value from ProcessInfo::getMemSizeMB() instead of getSystemMemSizeMB() (or from min(getSystemMemSizeMB(), getMemSizeMB())). Explicitly configured absolute values keep their current meaning. The IDL description and documentation should say "available memory", and a unit test with a mocked memLimit would cover the container case. 

        1. mem1.png
          mem1.png
          190 kB
        2. mem2.png
          mem2.png
          593 kB
        3. mem and threshold.png
          mem and threshold.png
          1.39 MB

            Assignee:
            Unassigned
            Reporter:
            逸帆 华 (EXT)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: