[SERVER-78049] Track query stats on changeStream and virtual collections Created: 13/Jun/23  Updated: 29/Oct/23  Resolved: 24/Aug/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.1.0-rc0

Type: Task Priority: Major - P3
Reporter: Maddie Zechar Assignee: Will Buerger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-85075 Tracking: Milestone 2 Closed
Assigned Teams:
Query Optimization
Backwards Compatibility: Fully Compatible
Sprint: QO 2023-06-26, QO 2023-07-10, QO 2023-07-24, QO 2023-08-07, QO 2023-08-21
Participants:

 Description   

Currently, running this $changeStreams test on master branch with no changes, fails on the getTelemetry() call with the error, "was not able to re-parse queryStats key when reading queryStats." Despite the original query being coll.aggregate({$changeStream: {}}, {$addFields: {newField: 1}}), the resulting queryShape is 

 

 { cmdNs: { db: "test", coll: "coll" }, command: "aggregate", pipeline: [ { $_internalChangeStreamOplogMatch: { filter: { $and: [ { ts: { $gte: "?timestamp" } }, { fromMigrate: { $not: { $eq: "?bool" } } }, { $or: [ { $and: [ { $or: [ { $and: [ { o.to: { $eq: "?string" } }, { o.renameCollection: { $exists: "?bool" } } ] }, { o.drop: { $eq: "?string" } }, { o.renameCollection: { $eq: "?string" } } ] }, { ns: { $eq: "?string" } }, { op: { $eq: "?string" } } ] }, { $and: [ { $or: [ { o2.type: { $in: "?array<?string>" } }, { o2.migrateChunkToNewShard: { $exists: "?bool" } }, { o2.refineCollectionShardKey: { $exists: "?bool" } }, { o2.reshardBegin: { $exists: "?bool" } }, { o2.reshardCollection: { $exists: "?bool" } }, { o2.reshardDoneCatchUp: { $exists: "?bool" } }, { o2.shardCollection: { $exists: "?bool" } } ] }, { op: { $eq: "?string" } }, { ns: { $regex: "?string" } } ] }, { $and: [ { ns: { $regex: "?string" } }, { $nor: [ { op: { $eq: "?string" } }, { op: { $eq: "?string" } } ] } ] }, { $and: [ { $or: [ { o.drop: { $regex: "?string" } }, { o.renameCollection: { $regex: "?string" } }, { $and: [ { o.to: { $regex: "?string" } }, { o.renameCollection: { $exists: "?bool" } } ] }, { o.create: { $regex: "?string" } }, { o.createIndexes: { $regex: "?string" } }, { o.commitIndexBuild: { $regex: "?string" } }, { o.dropIndexes: { $regex: "?string" } }, { o.collMod: { $regex: "?string" } } ] }, { op: { $eq: "?string" } }, { ns: { $regex: "?string" } } ] }, { $and: [ { $or: [ { o.applyOps: { $elemMatch: { $and: [ { $or: [ { o.create: { $regex: "?string" } }, { o.createIndexes: { $regex: "?string" } } ] }, { ns: { $regex: "?string" } } ] } } }, { o.applyOps.ns: { $regex: "?string" } }, { prevOpTime: { $not: { $eq: "?object" } } } ] }, { op: { $eq: "?string" } }, { o.partialTxn: { $not: { $eq: "?bool" } } }, { o.prepare: { $not: { $eq: "?bool" } } }, { o.applyOps: { $type: "?array<?number>" } } ] }, { $and: [ { o.commitTransaction: { $eq: "?number" } }, { op: { $eq: "?string" } } ] } ] } ] } } }, { $_internalChangeStreamUnwindTransaction: { filter: { $and: [ { $or: [ { $and: [ { $or: [ { $and: [ { o.to: { $regex: "?string" } }, { o.renameCollection: { $exists: "?bool" } } ] }, { o.collMod: { $regex: "?string" } }, { o.commitIndexBuild: { $regex: "?string" } }, { o.create: { $regex: "?string" } }, { o.createIndexes: { $regex: "?string" } }, { o.drop: { $regex: "?string" } }, { o.dropIndexes: { $regex: "?string" } }, { o.renameCollection: { $regex: "?string" } } ] }, { op: { $eq: "?string" } }, { ns: { $regex: "?string" } } ] }, { $and: [ { ns: { $regex: "?string" } }, { $nor: [ { op: { $eq: "?string" } }, { op: { $eq: "?string" } } ] } ] } ] }, { fromMigrate: { $not: { $eq: "?bool" } } } ] } } }, { $_internalChangeStreamTransform: { startAtOperationTime: "?timestamp", fullDocument: "default", fullDocumentBeforeChange: "off" } }, { $_internalChangeStreamCheckInvalidate: {} }, { $_internalChangeStreamCheckResumability: { resumeToken: "?object" } }, { $_internalChangeStreamCheckTopologyChange: {} }, { $_internalChangeStreamHandleTopologyChange: {} }, { $match: { operationType: { $in: "?array<?string>" } } }, { $addFields: { newField: "?number" } } ] } 

I pulled and rebase and rebuilt and the problem persists, so it doesn't seem to be the regex bug fix that got merged this week. 

 

So in other words, we need to bug fix tracking query stats for $changeStream and also log that into the collectionType field of the shape.

 

We should be able to easily track if it's a changeStream collection type by doing, `liteParsedPipeline.hasChangeStream()` where liteParsedPipeline is an existing var inside runAggregate



 Comments   
Comment by Githook User [ 23/Aug/23 ]

Author:

{'name': 'Will Buerger', 'email': 'will.buerger@mongodb.com', 'username': 'wbuerger46'}

Message: SERVER-78049: Track query stats on changeStream and virtual collections
Branch: master
https://github.com/mongodb/mongo/commit/e99747c377acc1e4a8ca3b9ee2047f0047e9724e

Generated at Thu Feb 08 06:37:19 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.