Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-78049

Track query stats on changeStream and virtual collections

    XMLWordPrintableJSON

Details

    • Icon: Task Task
    • Resolution: Fixed
    • Icon: Major - P3 Major - P3
    • 7.1.0-rc0
    • None
    • None
    • None
    • Query Optimization
    • Fully Compatible
    • QO 2023-06-26, QO 2023-07-10, QO 2023-07-24, QO 2023-08-07, QO 2023-08-21

    Description

      Currently, running this $changeStreams test on master branch with no changes, fails on the getTelemetry() call with the error, "was not able to re-parse queryStats key when reading queryStats." Despite the original query being coll.aggregate({$changeStream: {}}, {$addFields: {newField: 1}}), the resulting queryShape is 

       

       { cmdNs: { db: "test", coll: "coll" }, command: "aggregate", pipeline: [ { $_internalChangeStreamOplogMatch: { filter: { $and: [ { ts: { $gte: "?timestamp" } }, { fromMigrate: { $not: { $eq: "?bool" } } }, { $or: [ { $and: [ { $or: [ { $and: [ { o.to: { $eq: "?string" } }, { o.renameCollection: { $exists: "?bool" } } ] }, { o.drop: { $eq: "?string" } }, { o.renameCollection: { $eq: "?string" } } ] }, { ns: { $eq: "?string" } }, { op: { $eq: "?string" } } ] }, { $and: [ { $or: [ { o2.type: { $in: "?array<?string>" } }, { o2.migrateChunkToNewShard: { $exists: "?bool" } }, { o2.refineCollectionShardKey: { $exists: "?bool" } }, { o2.reshardBegin: { $exists: "?bool" } }, { o2.reshardCollection: { $exists: "?bool" } }, { o2.reshardDoneCatchUp: { $exists: "?bool" } }, { o2.shardCollection: { $exists: "?bool" } } ] }, { op: { $eq: "?string" } }, { ns: { $regex: "?string" } } ] }, { $and: [ { ns: { $regex: "?string" } }, { $nor: [ { op: { $eq: "?string" } }, { op: { $eq: "?string" } } ] } ] }, { $and: [ { $or: [ { o.drop: { $regex: "?string" } }, { o.renameCollection: { $regex: "?string" } }, { $and: [ { o.to: { $regex: "?string" } }, { o.renameCollection: { $exists: "?bool" } } ] }, { o.create: { $regex: "?string" } }, { o.createIndexes: { $regex: "?string" } }, { o.commitIndexBuild: { $regex: "?string" } }, { o.dropIndexes: { $regex: "?string" } }, { o.collMod: { $regex: "?string" } } ] }, { op: { $eq: "?string" } }, { ns: { $regex: "?string" } } ] }, { $and: [ { $or: [ { o.applyOps: { $elemMatch: { $and: [ { $or: [ { o.create: { $regex: "?string" } }, { o.createIndexes: { $regex: "?string" } } ] }, { ns: { $regex: "?string" } } ] } } }, { o.applyOps.ns: { $regex: "?string" } }, { prevOpTime: { $not: { $eq: "?object" } } } ] }, { op: { $eq: "?string" } }, { o.partialTxn: { $not: { $eq: "?bool" } } }, { o.prepare: { $not: { $eq: "?bool" } } }, { o.applyOps: { $type: "?array<?number>" } } ] }, { $and: [ { o.commitTransaction: { $eq: "?number" } }, { op: { $eq: "?string" } } ] } ] } ] } } }, { $_internalChangeStreamUnwindTransaction: { filter: { $and: [ { $or: [ { $and: [ { $or: [ { $and: [ { o.to: { $regex: "?string" } }, { o.renameCollection: { $exists: "?bool" } } ] }, { o.collMod: { $regex: "?string" } }, { o.commitIndexBuild: { $regex: "?string" } }, { o.create: { $regex: "?string" } }, { o.createIndexes: { $regex: "?string" } }, { o.drop: { $regex: "?string" } }, { o.dropIndexes: { $regex: "?string" } }, { o.renameCollection: { $regex: "?string" } } ] }, { op: { $eq: "?string" } }, { ns: { $regex: "?string" } } ] }, { $and: [ { ns: { $regex: "?string" } }, { $nor: [ { op: { $eq: "?string" } }, { op: { $eq: "?string" } } ] } ] } ] }, { fromMigrate: { $not: { $eq: "?bool" } } } ] } } }, { $_internalChangeStreamTransform: { startAtOperationTime: "?timestamp", fullDocument: "default", fullDocumentBeforeChange: "off" } }, { $_internalChangeStreamCheckInvalidate: {} }, { $_internalChangeStreamCheckResumability: { resumeToken: "?object" } }, { $_internalChangeStreamCheckTopologyChange: {} }, { $_internalChangeStreamHandleTopologyChange: {} }, { $match: { operationType: { $in: "?array<?string>" } } }, { $addFields: { newField: "?number" } } ] } 

      I pulled and rebase and rebuilt and the problem persists, so it doesn't seem to be the regex bug fix that got merged this week. 

       

      So in other words, we need to bug fix tracking query stats for $changeStream and also log that into the collectionType field of the shape.

       

      We should be able to easily track if it's a changeStream collection type by doing, `liteParsedPipeline.hasChangeStream()` where liteParsedPipeline is an existing var inside runAggregate

      Attachments

        Activity

          People

            will.buerger@mongodb.com Will Buerger
            maddie.zechar@mongodb.com Maddie Zechar
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: