• Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: 7.0.16
    • Component/s: None
    • ALL
    • Hide

      Minimal functional repro: over-invalidation

      1. Deploy a 3-shard cluster, FCV 7.0.
      2. Create and shard a timeseries collection:
        // code placeholder
        db.createCollection("tsperf", {timeseries: {timeField: "t", metaField: "m", granularity: "minutes"}});
        sh.shardCollection("db.tsperf", {m: "hashed"});
        db.adminCommand({configureCollectionBalancing: "db.system.buckets.tsperf", chunkSize: 128});
        
      1. Split the bucket namespace into two chunks and record their ranges:
        // code placeholder
        db.getSiblingDB("config").chunks.find({ns: "db.system.buckets.tsperf"}, {min: 1, max: 1, shard: 1})
        
      1. Pick two meta values that fall into different chunks (`rangeA` ,`rangeZ` ) and insert a few thousand measurements for each, so both buckets are materialized and hot.
      2. Confirm they are live:`db.serverStatus().bucketCatalog` (`numBuckets` /`numOpenBuckets` ).
      3. Move only the chunk that contains`rangeA` :
        // code placeholder
        db.adminCommand({moveChunk: "db.system.buckets.tsperf", find: {meta: "rangeA"}, to: "<shard2>", _waitForDelete: true})
        
      1. Immediately insert a single measurement with`meta: "rangeZ"` — a key entirely outside the migrated chunk.
      2. Measure the deltas of`db.serverStatus().wiredTiger.cache["bytes read into cache"]` and`db.serverStatus().bucketCatalog.numBucketsReopened` across steps 6-7.

       

      Expected : the write to`rangeZ` uses its still-valid in-memory bucket handle; no disk read;`numBucketsReopened` unchanged.

      Actual :`rangeZ` 's handle was invalidated by the migration of an unrelated chunk; the insert triggers a bucket document lookup on disk (query-based reopening),`numBucketsReopened` increments and`bytes read into cache` grows.

      B. Performance repro at production scale

      1. Same cluster, keeping ~1.6M bucket handles live per node (sustained writes over ~50M active series, granularity=minutes).
      2. Run a steady background writer across the full key space.
      3. Trigger`moveChunk` of one chunk off the donor.
      4. On the donor, sample at 1s granularity:`wiredTiger.cache["bytes read into cache"]` ,`connections.current` ,`opcounters.command` , plus CPU.
      5. Observed in production: read 122 → 647 MB/s, connections 316 → 1190, command rate 833 → 7432, CPU saturation 60-90 seconds after commit and lasting 2-3 minutes.

       

      In production the migrations are driven by the hourly TTL cohort, but the TTL cycle is not required to reproduce — a manual`moveChunk` takes the same code path in`migration_source_manager.cpp` (clear on commit)。

       
      Show
      Minimal functional repro: over-invalidation Deploy a 3-shard cluster, FCV 7.0. Create and shard a timeseries collection: // code placeholder db.createCollection( "tsperf" , {timeseries: {timeField: "t" , metaField: "m" , granularity: "minutes" }}); sh.shardCollection( "db.tsperf" , {m: "hashed" }); db.adminCommand({configureCollectionBalancing: "db.system.buckets.tsperf" , chunkSize: 128}); Split the bucket namespace into two chunks and record their ranges: // code placeholder db.getSiblingDB( "config" ).chunks.find({ns: "db.system.buckets.tsperf" }, {min: 1, max: 1, shard: 1}) Pick two meta values that fall into different chunks (`rangeA` ,`rangeZ` ) and insert a few thousand measurements for each, so both buckets are materialized and hot. Confirm they are live:`db.serverStatus().bucketCatalog` (`numBuckets` /`numOpenBuckets` ). Move only the chunk that contains`rangeA` : // code placeholder db.adminCommand({moveChunk: "db.system.buckets.tsperf" , find: {meta: "rangeA" }, to: "<shard2>" , _waitForDelete: true }) Immediately insert a single measurement with`meta: "rangeZ"` — a key entirely outside the migrated chunk. Measure the deltas of`db.serverStatus().wiredTiger.cache ["bytes read into cache"] ` and`db.serverStatus().bucketCatalog.numBucketsReopened` across steps 6-7.   Expected : the write to`rangeZ` uses its still-valid in-memory bucket handle; no disk read;`numBucketsReopened` unchanged. Actual :`rangeZ` 's handle was invalidated by the migration of an unrelated chunk; the insert triggers a bucket document lookup on disk (query-based reopening),`numBucketsReopened` increments and`bytes read into cache` grows. B. Performance repro at production scale Same cluster, keeping ~1.6M bucket handles live per node (sustained writes over ~50M active series, granularity=minutes). Run a steady background writer across the full key space. Trigger`moveChunk` of one chunk off the donor. On the donor, sample at 1s granularity:`wiredTiger.cache ["bytes read into cache"] ` ,`connections.current` ,`opcounters.command` , plus CPU. Observed in production: read 122 → 647 MB/s, connections 316 → 1190, command rate 833 → 7432, CPU saturation 60-90 seconds after commit and lasting 2-3 minutes.   In production the migrations are driven by the hourly TTL cohort, but the TTL cycle is not required to reproduce — a manual`moveChunk` takes the same code path in`migration_source_manager.cpp` (clear on commit)。  
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Environment:

      • MongoDB 7.0.x, 32-shard sharded cluster.
      • Timeseries collection: granularity=minutes, bucketMaxSpanSeconds=86400, shard key {meta: "hashed"}, ~53.5M active series, sustained per-minute writes

      Symptom Every hour at :00, one or more shards saturate CPU for 2-3 minutes and write latency spikes. In a 24h window, 52 CPU-saturation episodes (>=90%) were observed, and 52/52 occurred on shards that were performing a moveChunk as donor. Of 561 donor migrations in that window, 154 happened in the hourly batch and 52 of those (33.8%) triggered saturation; the other 407 non-hourly migrations triggered none.

       

      Evidence:

      • The migrations are pinned to the top of the hour 24h distribution of moveChunk.start for this namespace, grouped by mm:ss:
      // code placeholder
      SECONDARY> db.getSiblingDB("config").changelog.aggregate([{$match: {what: "moveChunk.start", ns: "cn5.system.buckets.time_series",time: {$gte: new Date(Date.now() - 24*3600*1000)}}},{$project: {_id: 0, hhmm: {$dateToString: {format: "%M:%S", date: "$time", timezone: "+08:00"}}}},{$group: {_id: "$hhmm", n: {$sum: 1}}},{$sort: {n: -1}}, {$limit: 10}])
      { "_id" : "00:24", "n" : 7 }{ "_id" : "00:31", "n" : 5 }{ "_id" : "00:39", "n" : 4 }{ "_id" : "00:26", "n" : 3 }{ "_id" : "00:36", "n" : 3 }{ "_id" : "00:25", "n" : 3 }{ "_id" : "00:23", "n" : 3 }{ "_id" : "00:35", "n" : 3 }{ "_id" : "00:56", "n" : 2 }{ "_id" : "00:46", "n" : 2 }
      
      •  All events fall in minute 00; only the seconds vary (23-56). Over a 3h window there were 11 migrations, all at 05:00 / 06:00 / 07:00 UTC.
      • All three donors (s25, s26, s28) showed a CPU rise in that window.
      • // code placeholder
        SECONDARY> db.getSiblingDB("config").changelog.aggregate([{$match: {what: "moveChunk.start", ns: "cn5.system.buckets.time_series", time: {$gte: ISODate("2026-11-17T05:00:00Z"), $lt:  ISODate("2026-11-17T05:05:00Z")}}}, {$project: {_id: 0, time: 1,from: "$details.from",to:   "$details.to",min:  "$details.min",max:  "$details.max"}},{$sort: {time: 1}}])
        
        { "time" : ISODate("2026-11-17T05:00:24.714Z"), "from" : "...-s26", "to" : "...-s17",
          "min" : { "meta" : NumberLong("-9086261400347544590") }, "max" : { "meta" : NumberLong("-9085166404023334406") } }
        { "time" : ISODate("2026-11-17T05:00:24.753Z"), "from" : "...-s25", "to" : "...-s16",
          "min" : { "meta" : NumberLong("-9134789826904278286") }, "max" : { "meta" : NumberLong("-9133883862012138121") } }
        { "time" : ISODate("2026-11-17T05:00:39.857Z"), "from" : "...-s28", "to" : "...-s17",
          "min" : { "meta" : NumberLong("-9135873293975906565") }, "max" : { "meta" : NumberLong("-9135388802156901633") } }
        { "time" : ISODate("2026-11-17T05:00:39.899Z"), "from" : "...-s26", "to" : "...-s7",
          "min" : { "meta" : NumberLong("-9086261400347544590") }, "max" : { "meta" : NumberLong("-9085166404023334406") } }
        
      • The imbalance is produced by the hourly TTL deletion

      TTL delete rate in the :00 30-second bucket: receivers s17-1 = 2706.9, s16-0 = 2619.5, s7-1 = 1585 docs/s; all three donors (s25, s26, s28) = 0.

      2706.9 * 30s ~= 81k buckets * ~5120 B/bucket ~= 400 MB ~= 3 x 128 MB, i.e. exactly the balancer's imbalance threshold.

      Mechanism: ShardsvrGetStatsForBalancing reports avgObjSize * (numRecords - numOrphanDocs). TTL deletion lowers numRecords on the shards that delete fastest, so they become the lightest (receivers), while the shards that had not yet finished deleting become the heaviest (donors).

      • The spike is caused by the commit, not by the clone

      On donor s26-0, sampled at 30s granularity: during the clone phase CPU is flat. At commit, read throughput 122 -> 647 MB/s, connections 316 -> 1190, command rate 833 -> 7432. CPU saturates ~60-90s after commit and stays saturated for 2-3 minutes.

      • Relevant screenshots have been included in the attachments.
      • Note: All timestamps shown in the screenshots are in UTC+8.

      Root cause On migration commit,`migration_source_manager.cpp` executes:

       

      if (nss().isTimeseriesBucketsCollection()) {
          auto& bucketCatalog = timeseries::bucket_catalog::BucketCatalog::get(_opCtx);
          clear(bucketCatalog, nss().getTimeseriesViewNamespace());
      }
      

      `clear()` invalidates every bucket handle of the view namespace, not only the buckets whose keys fall inside the migrated chunk range`[min, max)` . All in-flight bucket handles on the donor become stale, so every subsequent insert has to reload its bucket from disk (query-based reopening). Measured on the donor during one commit: read throughput 122 -> 647 MB/s, connections 316 -> 1190, command rate 833 -> 7432, CPU saturation ~60-90s after commit.

      Why the migrations are hourly Bucket boundaries are rounded to the hour, so one hourly cohort expires together. TTL deletion lowers`numRecords` on the affected shards;`ShardsvrGetStatsForBalancing` reports`avgObjSize * (numRecords - numOrphanDocs)` , so those shards fall below the ideal per-shard size and become receivers, while the heaviest shards become donors (threshold`fromSize - toSize >= 3 * maxChunkSizeBytes` ). In one 30s bucket at :00, receivers deleted 2706.9 / 2619.5 / 1585 docs/s while all three donors deleted 0 – i.e. the TTL burst itself creates the imbalance that drives the migration.

      Requested fix Keep the correctness guarantee (a donor must not keep updating buckets that were migrated away), but scope the invalidation to the migrated key range instead of the whole collection.

      Today`ShouldClearFn` is`std::function<bool(const NamespaceString&)>` in 7.0 and`std::function<bool(const UUID&)>` in 8.0+, so it cannot express a key-range predicate; extending it to accept the chunk's`[min, max)` (or a per-bucket ownership/epoch check) would remove the read amplification.

       

      Related:

      • SERVER-106179 – same symptom reported on 7.0.18 / 8 shards (CPU 100%, WT cache read ~8GB/s, slow queries are query-based reopening); closed via support, no fix.
      • SERVER-107438 – generalized this clear to viewless collections (fixVersion 8.2.0-rc0); it widens the scope, it does not narrow it.
        Note: 8.0 and 8.3 still clear the whole collection (by collection UUID); no version provides key-range scoped invalidation.

       

       

        1. cpu usage 04-05UTC.png
          cpu usage 04-05UTC.png
          736 kB
        2. ttl 04-05UTC.png
          ttl 04-05UTC.png
          1.93 MB
        3. command ops 04-05UTC.png
          command ops 04-05UTC.png
          1.87 MB
        4. command ops-24h.png
          command ops-24h.png
          2.17 MB
        5. cpu usage-24h.png
          cpu usage-24h.png
          2.66 MB
        6. current connection 04-05UTC.png
          current connection 04-05UTC.png
          1.90 MB
        7. ops latency avg 04-05UTC.png
          ops latency avg 04-05UTC.png
          1.86 MB
        8. wt bytes read into chache 04-05UTC.png
          wt bytes read into chache 04-05UTC.png
          2.02 MB
        9. wt read concurrent  trans out-24h.png
          wt read concurrent trans out-24h.png
          2.06 MB
        10. wt read concurrent trans out 04-05UTC.png
          wt read concurrent trans out 04-05UTC.png
          1.88 MB
        11. wt read concurrents available 04-05UTC.png
          wt read concurrents available 04-05UTC.png
          1.82 MB
        12. wt_session-24h.png
          wt_session-24h.png
          2.83 MB

            Assignee:
            Unassigned
            Reporter:
            逸帆 华 (EXT)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: