-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: 7.0.16
-
Component/s: None
-
ALL
-
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Environment:
- MongoDB 7.0.x, 32-shard sharded cluster.
- Timeseries collection: granularity=minutes, bucketMaxSpanSeconds=86400, shard key {meta: "hashed"}, ~53.5M active series, sustained per-minute writes
Symptom Every hour at :00, one or more shards saturate CPU for 2-3 minutes and write latency spikes. In a 24h window, 52 CPU-saturation episodes (>=90%) were observed, and 52/52 occurred on shards that were performing a moveChunk as donor. Of 561 donor migrations in that window, 154 happened in the hourly batch and 52 of those (33.8%) triggered saturation; the other 407 non-hourly migrations triggered none.
Evidence:
- The migrations are pinned to the top of the hour 24h distribution of moveChunk.start for this namespace, grouped by mm:ss:
// code placeholder SECONDARY> db.getSiblingDB("config").changelog.aggregate([{$match: {what: "moveChunk.start", ns: "cn5.system.buckets.time_series",time: {$gte: new Date(Date.now() - 24*3600*1000)}}},{$project: {_id: 0, hhmm: {$dateToString: {format: "%M:%S", date: "$time", timezone: "+08:00"}}}},{$group: {_id: "$hhmm", n: {$sum: 1}}},{$sort: {n: -1}}, {$limit: 10}]) { "_id" : "00:24", "n" : 7 }{ "_id" : "00:31", "n" : 5 }{ "_id" : "00:39", "n" : 4 }{ "_id" : "00:26", "n" : 3 }{ "_id" : "00:36", "n" : 3 }{ "_id" : "00:25", "n" : 3 }{ "_id" : "00:23", "n" : 3 }{ "_id" : "00:35", "n" : 3 }{ "_id" : "00:56", "n" : 2 }{ "_id" : "00:46", "n" : 2 }
- All events fall in minute 00; only the seconds vary (23-56). Over a 3h window there were 11 migrations, all at 05:00 / 06:00 / 07:00 UTC.
- All three donors (s25, s26, s28) showed a CPU rise in that window.
// code placeholder SECONDARY> db.getSiblingDB("config").changelog.aggregate([{$match: {what: "moveChunk.start", ns: "cn5.system.buckets.time_series", time: {$gte: ISODate("2026-11-17T05:00:00Z"), $lt: ISODate("2026-11-17T05:05:00Z")}}}, {$project: {_id: 0, time: 1,from: "$details.from",to: "$details.to",min: "$details.min",max: "$details.max"}},{$sort: {time: 1}}]) { "time" : ISODate("2026-11-17T05:00:24.714Z"), "from" : "...-s26", "to" : "...-s17", "min" : { "meta" : NumberLong("-9086261400347544590") }, "max" : { "meta" : NumberLong("-9085166404023334406") } } { "time" : ISODate("2026-11-17T05:00:24.753Z"), "from" : "...-s25", "to" : "...-s16", "min" : { "meta" : NumberLong("-9134789826904278286") }, "max" : { "meta" : NumberLong("-9133883862012138121") } } { "time" : ISODate("2026-11-17T05:00:39.857Z"), "from" : "...-s28", "to" : "...-s17", "min" : { "meta" : NumberLong("-9135873293975906565") }, "max" : { "meta" : NumberLong("-9135388802156901633") } } { "time" : ISODate("2026-11-17T05:00:39.899Z"), "from" : "...-s26", "to" : "...-s7", "min" : { "meta" : NumberLong("-9086261400347544590") }, "max" : { "meta" : NumberLong("-9085166404023334406") } }
- The imbalance is produced by the hourly TTL deletion
TTL delete rate in the :00 30-second bucket: receivers s17-1 = 2706.9, s16-0 = 2619.5, s7-1 = 1585 docs/s; all three donors (s25, s26, s28) = 0.
2706.9 * 30s ~= 81k buckets * ~5120 B/bucket ~= 400 MB ~= 3 x 128 MB, i.e. exactly the balancer's imbalance threshold.
Mechanism: ShardsvrGetStatsForBalancing reports avgObjSize * (numRecords - numOrphanDocs). TTL deletion lowers numRecords on the shards that delete fastest, so they become the lightest (receivers), while the shards that had not yet finished deleting become the heaviest (donors).
- The spike is caused by the commit, not by the clone
On donor s26-0, sampled at 30s granularity: during the clone phase CPU is flat. At commit, read throughput 122 -> 647 MB/s, connections 316 -> 1190, command rate 833 -> 7432. CPU saturates ~60-90s after commit and stays saturated for 2-3 minutes.
- Relevant screenshots have been included in the attachments.
- Note: All timestamps shown in the screenshots are in UTC+8.
Root cause On migration commit,`migration_source_manager.cpp` executes:
if (nss().isTimeseriesBucketsCollection()) {
auto& bucketCatalog = timeseries::bucket_catalog::BucketCatalog::get(_opCtx);
clear(bucketCatalog, nss().getTimeseriesViewNamespace());
}
`clear()` invalidates every bucket handle of the view namespace, not only the buckets whose keys fall inside the migrated chunk range`[min, max)` . All in-flight bucket handles on the donor become stale, so every subsequent insert has to reload its bucket from disk (query-based reopening). Measured on the donor during one commit: read throughput 122 -> 647 MB/s, connections 316 -> 1190, command rate 833 -> 7432, CPU saturation ~60-90s after commit.
Why the migrations are hourly Bucket boundaries are rounded to the hour, so one hourly cohort expires together. TTL deletion lowers`numRecords` on the affected shards;`ShardsvrGetStatsForBalancing` reports`avgObjSize * (numRecords - numOrphanDocs)` , so those shards fall below the ideal per-shard size and become receivers, while the heaviest shards become donors (threshold`fromSize - toSize >= 3 * maxChunkSizeBytes` ). In one 30s bucket at :00, receivers deleted 2706.9 / 2619.5 / 1585 docs/s while all three donors deleted 0 – i.e. the TTL burst itself creates the imbalance that drives the migration.
Requested fix Keep the correctness guarantee (a donor must not keep updating buckets that were migrated away), but scope the invalidation to the migrated key range instead of the whole collection.
Today`ShouldClearFn` is`std::function<bool(const NamespaceString&)>` in 7.0 and`std::function<bool(const UUID&)>` in 8.0+, so it cannot express a key-range predicate; extending it to accept the chunk's`[min, max)` (or a per-bucket ownership/epoch check) would remove the read amplification.
Related:
SERVER-106179– same symptom reported on 7.0.18 / 8 shards (CPU 100%, WT cache read ~8GB/s, slow queries are query-based reopening); closed via support, no fix.SERVER-107438– generalized this clear to viewless collections (fixVersion 8.2.0-rc0); it widens the scope, it does not narrow it.
Note: 8.0 and 8.3 still clear the whole collection (by collection UUID); no version provides key-range scoped invalidation.
- is related to
-
SERVER-139633 timeseriesIdleBucketExpiryMemoryUsageThreshold is derived from host total memory (getSystemMemSizeMB) instead of the cgroup limit, so idle bucket expiry never triggers in containers
-
- Needs Verification
-
- related to
-
SERVER-106179 Unexpected High CPU and WiredTiger Cache Load in Sharded Cluster Using TimeSeries Collections
-
- Closed
-
-
SERVER-107438 Clear bucket catalog on chunk migration commit for viewless timeseries collection
-
- Closed
-