• Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: 9.0.4, 9.0.8
    • Component/s: Query Execution, Write Ops
    • None
    • Environment:
      Single-node replica set, FCV 9.0, official mongo:9.0.8 Docker image (also seen on 9.0.4). Not sharded. Fixed one-day buckets (bucketMaxSpanSeconds = bucketRoundingSeconds = 86400).
    • Query Execution
    • ALL
    • Hide

      Run with mongosh <uri> repro.js. It usually fails within a few hundred trials. On 9.0.8: 3 of 300 trials failed without the hint, 0 of 300 with HINT = true. Removing the s:1, w:1 index also gives 0 failures.

      const HINT = false; // set true -> no failures
      const d = db.getSiblingDB("ts_delete_repro"); d.dropDatabase();
      d.createCollection("p", {timeseries: {timeField: "t", metaField: "s",
        bucketMaxSpanSeconds: 86400, bucketRoundingSeconds: 86400}});
      d.p.createIndex({s: 1, w: 1});
      d.p.createIndex({s: 1, m: 1});
      const base = ISODate("2024-03-15T00:00:00Z").getTime();
      const rnd = n => Math.floor(Math.random() * n);
      const S = ObjectId("65f000000000000000001001");
      let failures = 0;
      for (let trial = 0; trial < 300; trial++) {
        d.p.deleteMany({s: S});
        const expected = new Map(); // t -> w
        for (let w = 1; w <= 6; w++) {
          // insert a new version (w) at 1-8 random 30-minute slots over 3 days
          const ts = [...new Set(Array.from({length: 1 + rnd(8)}, () => base + rnd(144) * 1800000))];
          d.p.insertMany(ts.map(t => ({s: S, t: new Date(t), w: NumberLong(String(w)), v: 1.0, m: new Date()})),
            {writeConcern: {w: "majority"}});
          // delete older versions at exactly those timestamps
          const wantDeleted = ts.filter(t => expected.has(t)).length;
          const opts = {writeConcern: {w: "majority"}};
          if (HINT) opts.hint = {s: 1, t: 1};
          const r = d.p.deleteMany({s: S, t: {$in: ts.map(t => new Date(t))}, w: {$lt: NumberLong(String(w))}}, opts);
          ts.forEach(t => expected.set(t, w));
          const left = d.p.countDocuments({s: S});
          if (r.deletedCount !== wantDeleted || left !== expected.size) {
            failures++;
            print(`trial ${trial} w=${w}: deletedCount=${r.deletedCount} expected=${wantDeleted}; remaining=${left} expected=${expected.size}`);
            break;
          }
        }
      }
      print(`failures: ${failures} / 300 (hint=${HINT})`);
      
      Show
      Run with mongosh <uri> repro.js . It usually fails within a few hundred trials. On 9.0.8: 3 of 300 trials failed without the hint, 0 of 300 with HINT = true . Removing the s:1, w:1 index also gives 0 failures. const HINT = false ; // set true -> no failures const d = db.getSiblingDB( "ts_delete_repro" ); d.dropDatabase(); d.createCollection( "p" , {timeseries: {timeField: "t" , metaField: "s" , bucketMaxSpanSeconds: 86400, bucketRoundingSeconds: 86400}}); d.p.createIndex({s: 1, w: 1}); d.p.createIndex({s: 1, m: 1}); const base = ISODate( "2024-03-15T00:00:00Z" ).getTime(); const rnd = n => Math .floor( Math .random() * n); const S = ObjectId( "65f000000000000000001001" ); let failures = 0; for (let trial = 0; trial < 300; trial++) { d.p.deleteMany({s: S}); const expected = new Map(); // t -> w for (let w = 1; w <= 6; w++) { // insert a new version (w) at 1-8 random 30-minute slots over 3 days const ts = [... new Set(Array.from({length: 1 + rnd(8)}, () => base + rnd(144) * 1800000))]; d.p.insertMany(ts.map(t => ({s: S, t: new Date(t), w: NumberLong( String (w)), v: 1.0, m: new Date()})), {writeConcern: {w: "majority" }}); // delete older versions at exactly those timestamps const wantDeleted = ts.filter(t => expected.has(t)).length; const opts = {writeConcern: {w: "majority" }}; if (HINT) opts.hint = {s: 1, t: 1}; const r = d.p.deleteMany({s: S, t: {$ in : ts.map(t => new Date(t))}, w: {$lt: NumberLong( String (w))}}, opts); ts.forEach(t => expected.set(t, w)); const left = d.p.countDocuments({s: S}); if (r.deletedCount !== wantDeleted || left !== expected.size) { failures++; print( `trial ${trial} w=${w}: deletedCount=${r.deletedCount} expected=${wantDeleted}; remaining=${left} expected=${expected.size}` ); break ; } } } print( `failures: ${failures} / 300 (hint=${HINT})` );
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      On a time-series collection with a compound secondary index on meta, measurementField, a deleteMany with a filter like

      {s: X, t: {$in: [...]}, w: {$lt: N}}
      

      sometimes deletes measurements that do not match the filter. Sometimes deletedCount is correct while other measurements of the same series still disappear. Sometimes deletedCount is also too high (e.g. reported 9, expected 1; 32 of 39 measurements lost).

      The same workload never loses data when:

      • the delete is hinted to s: 1, t: 1, or
      • the s: 1, w: 1 index does not exist.

      So the problem seems specific to the delete path when it is planned on the measurement-field secondary index (bucket unpacking / residual predicate evaluation for that plan).

      Expected

      Each deleteMany removes only measurements with s == S, t in the list, and w < current. deletedCount equals the expected count and the remaining count matches on every step.

      Actual

      Occasionally (about 1% of trials) measurements outside the filter are deleted. Example from 9.0.8:

      trial 13 w=6 AFTER DELETE reported=9 wantDeleted=1 left=7 want=39
      

      The lost measurements were at timestamps not in the $in list, across all three daily buckets.

      Workaround

      Hint metaField: 1, timeField: 1 on deletes, or avoid secondary indexes on measurement fields that delete filters reference.

      Possibly related (none match)

            Assignee:
            [DO NOT USE] Backlog - Query Execution
            Reporter:
            Jarrah Watson (EXT)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: