[SERVER-47640] $lookup should increment the serverStatus metrics.queryExecutor counters Created: 17/Apr/20  Updated: 29/Oct/23  Resolved: 03/Jan/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 4.9.0

Type: Improvement Priority: Major - P3
Reporter: Chris Harris Assignee: Rishab Joshi (Inactive)
Resolution: Fixed Votes: 2
Labels: qexec-team
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Problem/Incident
Related
related to SERVER-53501 Incorrect scanned Objects in $unionWi... Closed
related to SERVER-53762 Report aggregate execution stats in e... Closed
is related to SERVER-53771 $facet should report its summary stats Closed
Backwards Compatibility: Fully Compatible
Sprint: Query 2020-05-18, Query 2020-11-30, Query 2020-12-14, Query 2020-12-28, Query 2021-01-11
Participants:
Case:
Linked BF Score: 14

 Description   

As demonstrated below, work performed by $lookup does not increment either the scanned or scannedObjects counters in serverStatus.  The result is that mongod does not accurately report the amount of work being performed in this area which complicates the diagnostic and troubleshooting process. 

 

In this example, we populate a local collection with one document and a foreign collection with 1,000 documents.  None of the work performed by the $lookup (including the full COLLSCAN of the 1,000 documents in the foreign collection) gets captured by the queryExecutor metrics.  

> db.local.drop()
false
> db.foreign.drop()
false
> 
> 
> db.local.insert({_id:0, localField:0})
WriteResult({ "nInserted" : 1 })
> for(i = 0; i < 1000; i++){db.foreign.insert({_id:i, foreignField:i})}
WriteResult({ "nInserted" : 1 })
> 
> 
> db.serverStatus().metrics.queryExecutor
{ "scanned" : NumberLong(0), "scannedObjects" : NumberLong(0) }
> 
> 
> db.local.find().itcount()
1
> db.serverStatus().metrics.queryExecutor
{ "scanned" : NumberLong(0), "scannedObjects" : NumberLong(1) }
> 
> 
> db.foreign.find().itcount()
1000
> db.serverStatus().metrics.queryExecutor
{ "scanned" : NumberLong(0), "scannedObjects" : NumberLong(1001) }
> 
> 
> db.local.aggregate([])
{ "_id" : 0, "localField" : 0 }
> db.serverStatus().metrics.queryExecutor
{ "scanned" : NumberLong(0), "scannedObjects" : NumberLong(1002) }
> 
> 
> db.local.aggregate([{$lookup:{from:'foreign', localField: 'localField', foreignField: 'foreignField', as: 'output'}}])
{ "_id" : 0, "localField" : 0, "output" : [ { "_id" : 0, "foreignField" : 0 } ] }
> db.serverStatus().metrics.queryExecutor
{ "scanned" : NumberLong(0), "scannedObjects" : NumberLong(1003) }
> 
> 
> 
> db.foreign.createIndex({foreignField:1})
{
	"createdCollectionAutomatically" : false,
	"numIndexesBefore" : 1,
	"numIndexesAfter" : 2,
	"ok" : 1
}
> db.foreign.find({foreignField:0})
{ "_id" : 0, "foreignField" : 0 }
> db.serverStatus().metrics.queryExecutor
{ "scanned" : NumberLong(1), "scannedObjects" : NumberLong(1004) }
> 
> 
> db.local.aggregate([{$lookup:{from:'foreign', localField: 'localField', foreignField: 'foreignField', as: 'output'}}])
{ "_id" : 0, "localField" : 0, "output" : [ { "_id" : 0, "foreignField" : 0 } ] }
> db.serverStatus().metrics.queryExecutor
{ "scanned" : NumberLong(1), "scannedObjects" : NumberLong(1005) }



 Comments   
Comment by Githook User [ 01/Jan/21 ]

Author:

{'name': 'Rishab Joshi', 'email': 'rishab.joshi@mongodb.com'}

Message: SERVER-47640 $lookup should increment the serverStatus metrics.queryExecutor counters
Branch: master
https://github.com/mongodb/mongo/commit/98a7731d21a8746e584f7092aadbee60a5fad6ef

Comment by Githook User [ 30/Dec/20 ]

Author:

{'name': 'Pierlauro Sciarelli', 'email': 'pierlauro.sciarelli@mongodb.com', 'username': 'pierlauro'}

Message: Revert "SERVER-47640 $lookup should increment the serverStatus metrics.queryExecutor counters"

This reverts commit 9df6363a6699dc20e3abbbd7de386798a4a6a524.
Branch: master
https://github.com/mongodb/mongo/commit/a933a819fa11163b764f0bad2ada0af266d26e1f

Comment by Githook User [ 30/Dec/20 ]

Author:

{'name': 'Rishab Joshi', 'email': 'rishab.joshi@mongodb.com'}

Message: SERVER-47640 $lookup should increment the serverStatus metrics.queryExecutor counters
Branch: master
https://github.com/mongodb/mongo/commit/9df6363a6699dc20e3abbbd7de386798a4a6a524

Comment by David Storch [ 23/Jul/20 ]

Flagging for re-triage, since Drew is now working on other priorities.

Generated at Thu Feb 08 05:14:48 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.