[SERVER-30882] Indicate when aggregation stages will be executed by mongos machinery in the explain output Created: 29/Aug/17  Updated: 06/Dec/22

Status: Backlog
Project: Core Server
Component/s: Aggregation Framework
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Charlie Swanson Assignee: Backlog - Query Optimization
Resolution: Unresolved Votes: 0
Labels: explain, open_todo_in_code
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-30704 Use the AsyncResultsMerger to merge r... Closed
Assigned Teams:
Query Optimization
Sprint: Query 2017-10-02
Participants:

 Description   

As part of SERVER-30704 we will move execution from the pipeline into the mongos execution machinery (RouterExecStages). We should give a clear indication to users that these stages are not being forgotten, but are instead handled outside the pipeline somehow.



 Comments   
Comment by Charlie Swanson [ 10/May/22 ]

I think this ticket is resolved? Is there any example of known missing stages? Here's a snippet of a recent explain example:

[js_test:agg_sort] [jsTest] {
[js_test:agg_sort] [jsTest]     "serverInfo" : {
[js_test:agg_sort] [jsTest]             "host" : "ip-10-122-13-190",
[js_test:agg_sort] [jsTest]             "port" : 30025,
[js_test:agg_sort] [jsTest]             "version" : "6.1.0-alpha",
[js_test:agg_sort] [jsTest]             "gitVersion" : "unknown"
[js_test:agg_sort] [jsTest]     },
[js_test:agg_sort] [jsTest]     "serverParameters" : {
[js_test:agg_sort] [jsTest]             "internalQueryFacetBufferSizeBytes" : 104857600,
[js_test:agg_sort] [jsTest]             "internalQueryFacetMaxOutputDocSizeBytes" : 104857600,
[js_test:agg_sort] [jsTest]             "internalLookupStageIntermediateDocumentMaxSizeBytes" : 104857600,
[js_test:agg_sort] [jsTest]             "internalDocumentSourceGroupMaxMemoryBytes" : 104857600,
[js_test:agg_sort] [jsTest]             "internalQueryMaxBlockingSortMemoryUsageBytes" : 104857600,
[js_test:agg_sort] [jsTest]             "internalQueryProhibitBlockingMergeOnMongoS" : 0,
[js_test:agg_sort] [jsTest]             "internalQueryMaxAddToSetBytes" : 104857600,
[js_test:agg_sort] [jsTest]             "internalDocumentSourceSetWindowFieldsMaxMemoryBytes" : 104857600
[js_test:agg_sort] [jsTest]     },
[js_test:agg_sort] [jsTest]     "mergeType" : "mongos",
[js_test:agg_sort] [jsTest]     "splitPipeline" : {
[js_test:agg_sort] [jsTest]             "shardsPart" : [
[js_test:agg_sort] [jsTest]                     {
[js_test:agg_sort] [jsTest]                             "$sort" : {
[js_test:agg_sort] [jsTest]                                     "sortKey" : {
[js_test:agg_sort] [jsTest]                                             "_id" : 1
[js_test:agg_sort] [jsTest]                                     }
[js_test:agg_sort] [jsTest]                             }
[js_test:agg_sort] [jsTest]                     },
[js_test:agg_sort] [jsTest]                     {
[js_test:agg_sort] [jsTest]                             "$project" : {
[js_test:agg_sort] [jsTest]                                     "x" : true,
[js_test:agg_sort] [jsTest]                                     "y" : true,
[js_test:agg_sort] [jsTest]                                     "_id" : false
[js_test:agg_sort] [jsTest]                             }
[js_test:agg_sort] [jsTest]                     }
[js_test:agg_sort] [jsTest]             ],
[js_test:agg_sort] [jsTest]             "mergerPart" : [
[js_test:agg_sort] [jsTest]                     {
[js_test:agg_sort] [jsTest]                             "$mergeCursors" : {
[js_test:agg_sort] [jsTest]                                     "lsid" : {
[js_test:agg_sort] [jsTest]                                             "id" : UUID("3ac5dea9-0a92-4bc2-878b-dbc22fb1ef5c"),
[js_test:agg_sort] [jsTest]                                             "uid" : BinData(0,"47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=")
[js_test:agg_sort] [jsTest]                                     },
[js_test:agg_sort] [jsTest]                                     "sort" : {
[js_test:agg_sort] [jsTest]                                             "_id" : 1
[js_test:agg_sort] [jsTest]                                     },
[js_test:agg_sort] [jsTest]                                     "compareWholeSortKey" : false,
[js_test:agg_sort] [jsTest]                                     "tailableMode" : "normal",
[js_test:agg_sort] [jsTest]                                     "nss" : "test.sharded_agg_sort",
[js_test:agg_sort] [jsTest]                                     "allowPartialResults" : false,
[js_test:agg_sort] [jsTest]                                     "recordRemoteOpWaitTime" : false
[js_test:agg_sort] [jsTest]                             }
[js_test:agg_sort] [jsTest]                     },
[js_test:agg_sort] [jsTest]                     {
[js_test:agg_sort] [jsTest]                             "$group" : {
[js_test:agg_sort] [jsTest]                                     "_id" : "$x",
[js_test:agg_sort] [jsTest]                                     "firstY" : {
[js_test:agg_sort] [jsTest]                                             "$first" : "$y"
[js_test:agg_sort] [jsTest]                                     }
[js_test:agg_sort] [jsTest]                             }
[js_test:agg_sort] [jsTest]                     }
[js_test:agg_sort] [jsTest]             ]
[js_test:agg_sort] [jsTest]     },
[js_test:agg_sort] [jsTest]     "shards" : {
[js_test:agg_sort] [jsTest]             "agg_sort-rs1" : {
[js_test:agg_sort] [jsTest]                     "host" : "ip-10-122-13-190:30021",
[js_test:agg_sort] [jsTest]                     "queryPlanner" : {
[js_test:agg_sort] [jsTest]                             "namespace" : "test.sharded_agg_sort",
[js_test:agg_sort] [jsTest]                             "indexFilterSet" : false,
[js_test:agg_sort] [jsTest]                             "parsedQuery" : {
[js_test:agg_sort] [jsTest]
[js_test:agg_sort] [jsTest]                             },
[js_test:agg_sort] [jsTest]                             "queryHash" : "CD5A1BCB",
[js_test:agg_sort] [jsTest]                             "planCacheKey" : "CD5A1BCB",
[js_test:agg_sort] [jsTest]                             "optimizedPipeline" : true,
[js_test:agg_sort] [jsTest]                             "maxIndexedOrSolutionsReached" : false,
[js_test:agg_sort] [jsTest]                             "maxIndexedAndSolutionsReached" : false,
[js_test:agg_sort] [jsTest]                             "maxScansToExplodeReached" : false,
[js_test:agg_sort] [jsTest]                             "winningPlan" : {
[js_test:agg_sort] [jsTest]                                     "stage" : "PROJECTION_SIMPLE",
[js_test:agg_sort] [jsTest]                                     "transformBy" : {
[js_test:agg_sort] [jsTest]                                             "x" : true,
[js_test:agg_sort] [jsTest]                                             "y" : true,
[js_test:agg_sort] [jsTest]                                             "_id" : false
[js_test:agg_sort] [jsTest]                                     },
[js_test:agg_sort] [jsTest]                                     "inputStage" : {
[js_test:agg_sort] [jsTest]                                             "stage" : "SORT_KEY_GENERATOR",
[js_test:agg_sort] [jsTest]                                             "inputStage" : {
[js_test:agg_sort] [jsTest]                                                     "stage" : "FETCH",
[js_test:agg_sort] [jsTest]                                                     "inputStage" : {
[js_test:agg_sort] [jsTest]                                                             "stage" : "SHARDING_FILTER",
[js_test:agg_sort] [jsTest]                                                             "inputStage" : { 

 

We also recently resolved https://jira.mongodb.org/browse/SERVER-57383 which is related/potentially a duplicate.

Comment by Bernard Gorman [ 18/Oct/17 ]

Following some investigation, it turns out that the stages which are absorbed from the merge pipeline into RouterExecStages during execution on mongoS are still present in the explain output, whose format remains unchanged by SERVER-30704. This is because in the explain path, we serialize the full merge pipeline to the output builder before the point where the leading stages would ordinarily be popped off.

Since this isn't the regression we suspected it might be, and after discussing with charlie.swanson, I'm returning this to the backlog for now. Leaving it open because augmenting the existing explain output to additionally report when merge-side stages will be executed outside of the pipeline on mongoS might be a worthwhile improvement at some point in the future.

Generated at Thu Feb 08 04:25:19 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.