-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: 7.0.16
-
Component/s: None
-
Environment:- `MongoDB 7.0.16, FCV 7.0, sharded cluster (32 shards), container with cgroup v2 limit 32 GiB on a 1 TB host, no lxcfs.`
-
ALL
-
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Defect:
The default value of timeseriesIdleBucketExpiryMemoryUsageThreshold is computed from host total memory via ProcessInfo::getSystemMemSizeMB(). In a container that does not mount lxcfs, /proc/meminfo reports the host's MemTotal, so the threshold can be far larger than the memory the process is actually allowed to use. Idle bucket expiry then never fires, and the bucket catalog grows until the container is OOM-killed.
This is a per-call-site issue, not a defect in getSystemMemSizeMB(): that function's contract (host total memory) is correct, and hostInfo deliberately exposes both memSize and memLimit (SERVER-39966). The problem is that an allocation/reclamation threshold is derived from total memory instead of the effective limit – the class of bug described in SERVER-16571.
Code locations:
7.0 – src/mongo/db/timeseries/timeseries_global_options.cpp L36-L52:
value = ProcessInfo::getSystemMemSizeMB() * 25 * 1024;
The cgroup-aware counterpart already exists and is used for WiredTiger cache sizing: ProcessInfo::getMemSizeMB() returns memLimit, where memLimit = min(systemMemory, cgroupLimit) (processinfo_linux.cpp L621-L640, wiredtiger_util.cpp L586-L607).
Observed values
On shard s3 (primary), after several hours of continuous writes:
PRIMARY> db.adminCommand({getParameter: 1, timeseriesIdleBucketExpiryMemoryUsageThreshold: 1})
{ "timeseriesIdleBucketExpiryMemoryUsageThreshold" : NumberLong("26394163200"), "ok" : 1 }
PRIMARY> db.serverStatus().bucketCatalog
{
"numBuckets" : 2618944,
"numOpenBuckets" : 2617510,
"numIdleBuckets" : 2617510,
"memoryUsage" : NumberLong("4432206461"),
"numBucketsClosedDueToMemoryThreshold" : 0,
"numBucketsArchivedDueToMemoryThreshold" : 0,
"numBucketsClosedDueToCachePressure" : 2007289,
...
}
The automatic threshold is 26,394,163,200 B, which is 25/1024 of 1,031,022 MB of host memory. This host runs in a container with a 32 GiB cgroup limit, so a threshold derived from the effective limit would be 838,860,800 B (800 MiB). The current value is 31.5x larger.
At the same moment:
- memoryUsage is 4,432,206,461 B, i.e. 5.3x the threshold a correct derivation would produce;
- numOpenBuckets equals numIdleBuckets (2,617,510), so the entire open set is idle and eligible for expiry;
- numBucketsClosedDueToMemoryThreshold and numBucketsArchivedDueToMemoryThreshold are both 0, so the memory-based expiry path has never run once.
The catalog has grown more than 5x past the limit a correct threshold would impose, every open bucket is idle, and not a single bucket has been reclaimed. This is not a leak; the threshold is unreachable.
Why this is a bug and not by design
SERVER-16571established that allocation decisions must use the actual memory constraint, "for example when running MongoDB within containers". That fix covered WiredTiger cache, oplog sizing and tcmalloc; this threshold, introduced later inSERVER-61195(5.0), was never added to that list.SERVER-95511states the goal as "a percentage that scales with the cluster", yet the implementation still reads host memory, so the percentage does not follow the container's size. On a 1 TB host with a 32 GiB limit it barely changes when the pod is resized.- The parameter is undocumented (see
SERVER-95510), so "set it explicitly" is not a reasonable answer for the default.
Impact
The container limit is 32 GiB and the WiredTiger cache already claims 15.5 GiB of it, while the catalog's automatic threshold is 24.6 GiB (26,394,163,200 B) – together well past the limit before connections, indexes and query memory.
Since the memory-based expiry path has never run (numBucketsClosedDueToMemoryThreshold = 0), growth is unbounded until the container is OOM-killed. This applies to any pod whose limit is a small fraction of host memory.
Suggested fix
Derive the automatic value from ProcessInfo::getMemSizeMB() instead of getSystemMemSizeMB() (or from min(getSystemMemSizeMB(), getMemSizeMB())). Explicitly configured absolute values keep their current meaning. The IDL description and documentation should say "available memory", and a unit test with a mocked memLimit would cover the container case.
- is related to
-
SERVER-106179 Unexpected High CPU and WiredTiger Cache Load in Sharded Cluster Using TimeSeries Collections
-
- Closed
-
-
SERVER-95510 Increase default bucket catalog memory usage
-
- Closed
-
- related to
-
SERVER-16571 Use Actual Memory Constraint vs. Total System Memory When They Differ
-
- Closed
-
-
SERVER-139631 Timeseries bucket catalog is cleared for the entire collection on chunk migration commit, causing read amplification and CPU saturation on the donor shard
-
- Needs Verification
-
-
SERVER-39966 Report both total actual system memory and cgroup memory limit in hostInfo
-
- Closed
-
-
SERVER-61195 Scale BucketCatalog memory limit based on RAM size
-
- Closed
-
-
SERVER-95511 Modify Time Series Collection Parameters to Support Autoscaling
-
- Closed
-