Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-29421

Aggregation executionStats or allPlansExecution explain modes behave incorrectly with $limit

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major - P3
    • Resolution: Fixed
    • 3.5.8
    • 3.7.2
    • Aggregation Framework
    • Fully Compatible
    • ALL
    • Query 2017-06-19, Query 2017-08-21, Query 2017-12-18, Query 2018-01-01, Query 2018-01-15, Query 2018-01-29

    Description

      When there is a limit that can be absorbed into the $cursor stage of the pipeline, this limit is not respected when reporting explain execution stats. Consider the following example:

      > db.c.drop()
      true
      > db.c.createIndex({a: 1})
      {
      	"createdCollectionAutomatically" : true,
      	"numIndexesBefore" : 1,
      	"numIndexesAfter" : 2,
      	"ok" : 1
      }
      > for (var i = 0; i < 10; i++) { db.c.insert({a: 1}); }
      WriteResult({ "nInserted" : 1 })
      > db.c.explain("executionStats").aggregate([{$match: {a: 1}}, {$limit: 3}])
      {
      	"stages" : [
      		{
      			"$cursor" : {
      				"query" : {
      					"a" : 1
      				},
      				"limit" : NumberLong(3),
      				"queryPlanner" : {
      					"plannerVersion" : 1,
      					"namespace" : "test.c",
      					"indexFilterSet" : false,
      					"parsedQuery" : {
      						"a" : {
      							"$eq" : 1
      						}
      					},
      					"winningPlan" : {
      						"stage" : "FETCH",
      						"inputStage" : {
      							"stage" : "IXSCAN",
      							"keyPattern" : {
      								"a" : 1
      							},
      							"indexName" : "a_1",
      							"isMultiKey" : false,
      							"multiKeyPaths" : {
      								"a" : [ ]
      							},
      							"isUnique" : false,
      							"isSparse" : false,
      							"isPartial" : false,
      							"indexVersion" : 2,
      							"direction" : "forward",
      							"indexBounds" : {
      								"a" : [
      									"[1.0, 1.0]"
      								]
      							}
      						}
      					},
      					"rejectedPlans" : [ ]
      				},
      				"executionStats" : {
      					"executionSuccess" : true,
      					"nReturned" : 10,
      					"executionTimeMillis" : 3,
      					"totalKeysExamined" : 10,
      					"totalDocsExamined" : 10,
      					"executionStages" : {
      						"stage" : "FETCH",
      						"nReturned" : 10,
      						"executionTimeMillisEstimate" : 0,
      						"works" : 11,
      						"advanced" : 10,
      						"needTime" : 0,
      						"needYield" : 0,
      						"saveState" : 1,
      						"restoreState" : 1,
      						"isEOF" : 1,
      						"invalidates" : 0,
      						"docsExamined" : 10,
      						"alreadyHasObj" : 0,
      						"inputStage" : {
      							"stage" : "IXSCAN",
      							"nReturned" : 10,
      							"executionTimeMillisEstimate" : 0,
      							"works" : 11,
      							"advanced" : 10,
      							"needTime" : 0,
      							"needYield" : 0,
      							"saveState" : 1,
      							"restoreState" : 1,
      							"isEOF" : 1,
      							"invalidates" : 0,
      							"keyPattern" : {
      								"a" : 1
      							},
      							"indexName" : "a_1",
      							"isMultiKey" : false,
      							"multiKeyPaths" : {
      								"a" : [ ]
      							},
      							"isUnique" : false,
      							"isSparse" : false,
      							"isPartial" : false,
      							"indexVersion" : 2,
      							"direction" : "forward",
      							"indexBounds" : {
      								"a" : [
      									"[1.0, 1.0]"
      								]
      							},
      							"keysExamined" : 10,
      							"seeks" : 1,
      							"dupsTested" : 0,
      							"dupsDropped" : 0,
      							"seenInvalidated" : 0
      						}
      					}
      				}
      			}
      		}
      	],
      	"ok" : 1
      }
      

      Here the limit of 3 is indeed present in the $cursor.limit field. However, the $cursor.executionStats field shows that the index scan stage examined all 10 keys and that the fetch stage subsequently retrieved all 10 documents. This reporting is incorrect, since the limit of 3 will prevent us from having to look at all matching keys/documents.

      The root cause of this problem is that the limit is applied here as the DocumentSourceCursor is retrieving results from the underlying PlanExecutor. However, this code is not exercised by the explain path. Instead, when running an explain we execute the plan and retrieve stats here. Note that this only affects versions 3.5.5 and newer, e.g. versions after execution stats were added to aggregation explain in SERVER-19758.

      To fix this, we should have the DocumentSourceCursor to push the limit down into the PlanExecutor itself. This is also wise from an architecture point of view, since DocumentSourceCursor is intended to be a wrapper around a PlanExecutor rather than acting as a query execution operator in its own right.

      Attachments

        Issue Links

          Activity

            People

              ian.boros@mongodb.com Ian Boros
              david.storch@mongodb.com David Storch
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: