[SERVER-69056] 'explain' should include actual resource consumption costs when run with verbosity 'executionStats' Created: 22/Aug/22  Updated: 29/Oct/23  Resolved: 18/Nov/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 6.3.0-rc0

Type: Improvement Priority: Major - P3
Reporter: Eric Milkie Assignee: Adi Agrawal
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Documented
is documented by DOCS-15749 [SERVER] Investigate changes in SERVE... Backlog
Backwards Compatibility: Fully Compatible
Sprint: QE 2022-11-14, QE 2022-11-28
Participants:

 Description   

Original title: 'explain' should include estimated resource consumption costs
In order to make it easier for MongoDB users to tally resource consumption costs for Serverless, it would be useful for resource consumption to be returned as part of the 'explain' output.

Example:

test> db.foo.find({}).explain()
{
  explainVersion: '1',
  queryPlanner: {
    namespace: 'test.foo',
    indexFilterSet: false,
    parsedQuery: {},
    queryHash: '17830885',
    planCacheKey: '17830885',
    maxIndexedOrSolutionsReached: false,
    maxIndexedAndSolutionsReached: false,
    maxScansToExplodeReached: false,
    winningPlan: { stage: 'COLLSCAN', direction: 'forward' },
    rejectedPlans: []
  },
  command: { find: 'foo', filter: {}, '$db': 'test' },
  operationMetrics: { docBytesRead: 51, docUnitsRead:2 }

Actual implementation:

When explain is run with verbosity executionStats or higher, the "executionStats" now contain "operationMetrics" as well if profileOperationResourceConsumptionMetrics is enabled. These operationMetrics show runtime stats, if they are not zero. The stats are nested within "operationMetrics" which is nested in "executionStats." They include:

"cpuNanos"
"cursorSeeks"
"docBytesRead"
"docBytesWritten"
"docUnitsRead"
"docUnitsReturned"
"docUnitsWritten"
"idxEntryBytesRead"
"idxEntryBytesWritten"
"idxEntryUnitsRead"
"idxEntryUnitsWritten"
"totalUnitsWritten"
"keysSorted"
"sorterSpills"



 Comments   
Comment by Githook User [ 18/Nov/22 ]

Author:

{'name': 'Adityavardhan Agrawal', 'email': 'aa729@cornell.edu', 'username': 'Adityav369'}

Message: SERVER-69056 Show resource consumption metrics in explain
Branch: master
https://github.com/mongodb/mongo/commit/5765010714b9aa348be53b01a60fd4659216f53f

Comment by Kyle Suarez [ 28/Oct/22 ]

On discussion with bernard.gorman@mongodb.com at our director triage:

  • We have updated the title of this ticket to reflect that we want explain to show the actual resource consumption costs (there is currently no way to get any sort of estimate now)
  • We should already be reporting these usage stats in the logs (need to enable --setParameter profileOperationResourceConsumptionMetrics=1)
  • It's unclear if the stats are available to the query system at explain time, or if they're finalized when creating the command response. Seems straightforward to make the stats accessible to explain, though.

Removing this from director triage and assigning to myself to find an assignee.

Comment by Ana Meza [ 25/Oct/22 ]

bernard.gorman@mongodb.com kyle.suarez@mongodb.com could you assign this to someone?

Comment by Will Huang [ 24/Oct/22 ]

Hi team, it would be helpful for serverless usage investigations to have this option available when running the explain command for a given query. This would be quite useful for customer's serverless usage investigation related cases.

Comment by Eric Milkie [ 12/Sep/22 ]

It would look similar as the example above; the operationMetrics field would be at the top level. Ideally the non-winning plans that the command attempts would not be included in this.

brenda.rodriguez@mongodb.com I would say this is required for Serverless.

Comment by Vishal Dhiman (Inactive) [ 09/Sep/22 ]

milkie@mongodb.com Yes, this will be very useful. What will the output look like for IXSCAN?

 

Comment by Eric Milkie [ 09/Sep/22 ]

vishal.dhiman@mongodb.com do you think this is a feature we should include for Serverless? I can imagine users taking advantage of this to know how much an individual query can cost. I don't think there is currently any other way for a user to see this information.

Comment by Brenda Rodriguez [ 06/Sep/22 ]

milkie@mongodb.com and kateryna.kamenieva@mongodb.com could you please let us know if this is required for server less and belongs in an epic or if it's just a "nice to have?" Thanks!

Generated at Thu Feb 08 06:12:28 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.