Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-29421

Aggregation executionStats or allPlansExecution explain modes behave incorrectly with $limit

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: 3.5.8
    • Fix Version/s: 3.7.2
    • Component/s: Aggregation Framework
    • Labels:
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Sprint:
      Query 2017-06-19, Query 2017-08-21, Query 2017-12-18, Query 2018-01-01, Query 2018-01-15, Query 2018-01-29
    • Case:

      Description

      When there is a limit that can be absorbed into the $cursor stage of the pipeline, this limit is not respected when reporting explain execution stats. Consider the following example:

      > db.c.drop()
      true
      > db.c.createIndex({a: 1})
      {
      	"createdCollectionAutomatically" : true,
      	"numIndexesBefore" : 1,
      	"numIndexesAfter" : 2,
      	"ok" : 1
      }
      > for (var i = 0; i < 10; i++) { db.c.insert({a: 1}); }
      WriteResult({ "nInserted" : 1 })
      > db.c.explain("executionStats").aggregate([{$match: {a: 1}}, {$limit: 3}])
      {
      	"stages" : [
      		{
      			"$cursor" : {
      				"query" : {
      					"a" : 1
      				},
      				"limit" : NumberLong(3),
      				"queryPlanner" : {
      					"plannerVersion" : 1,
      					"namespace" : "test.c",
      					"indexFilterSet" : false,
      					"parsedQuery" : {
      						"a" : {
      							"$eq" : 1
      						}
      					},
      					"winningPlan" : {
      						"stage" : "FETCH",
      						"inputStage" : {
      							"stage" : "IXSCAN",
      							"keyPattern" : {
      								"a" : 1
      							},
      							"indexName" : "a_1",
      							"isMultiKey" : false,
      							"multiKeyPaths" : {
      								"a" : [ ]
      							},
      							"isUnique" : false,
      							"isSparse" : false,
      							"isPartial" : false,
      							"indexVersion" : 2,
      							"direction" : "forward",
      							"indexBounds" : {
      								"a" : [
      									"[1.0, 1.0]"
      								]
      							}
      						}
      					},
      					"rejectedPlans" : [ ]
      				},
      				"executionStats" : {
      					"executionSuccess" : true,
      					"nReturned" : 10,
      					"executionTimeMillis" : 3,
      					"totalKeysExamined" : 10,
      					"totalDocsExamined" : 10,
      					"executionStages" : {
      						"stage" : "FETCH",
      						"nReturned" : 10,
      						"executionTimeMillisEstimate" : 0,
      						"works" : 11,
      						"advanced" : 10,
      						"needTime" : 0,
      						"needYield" : 0,
      						"saveState" : 1,
      						"restoreState" : 1,
      						"isEOF" : 1,
      						"invalidates" : 0,
      						"docsExamined" : 10,
      						"alreadyHasObj" : 0,
      						"inputStage" : {
      							"stage" : "IXSCAN",
      							"nReturned" : 10,
      							"executionTimeMillisEstimate" : 0,
      							"works" : 11,
      							"advanced" : 10,
      							"needTime" : 0,
      							"needYield" : 0,
      							"saveState" : 1,
      							"restoreState" : 1,
      							"isEOF" : 1,
      							"invalidates" : 0,
      							"keyPattern" : {
      								"a" : 1
      							},
      							"indexName" : "a_1",
      							"isMultiKey" : false,
      							"multiKeyPaths" : {
      								"a" : [ ]
      							},
      							"isUnique" : false,
      							"isSparse" : false,
      							"isPartial" : false,
      							"indexVersion" : 2,
      							"direction" : "forward",
      							"indexBounds" : {
      								"a" : [
      									"[1.0, 1.0]"
      								]
      							},
      							"keysExamined" : 10,
      							"seeks" : 1,
      							"dupsTested" : 0,
      							"dupsDropped" : 0,
      							"seenInvalidated" : 0
      						}
      					}
      				}
      			}
      		}
      	],
      	"ok" : 1
      }
      

      Here the limit of 3 is indeed present in the $cursor.limit field. However, the $cursor.executionStats field shows that the index scan stage examined all 10 keys and that the fetch stage subsequently retrieved all 10 documents. This reporting is incorrect, since the limit of 3 will prevent us from having to look at all matching keys/documents.

      The root cause of this problem is that the limit is applied here as the DocumentSourceCursor is retrieving results from the underlying PlanExecutor. However, this code is not exercised by the explain path. Instead, when running an explain we execute the plan and retrieve stats here. Note that this only affects versions 3.5.5 and newer, e.g. versions after execution stats were added to aggregation explain in SERVER-19758.

      To fix this, we should have the DocumentSourceCursor to push the limit down into the PlanExecutor itself. This is also wise from an architecture point of view, since DocumentSourceCursor is intended to be a wrapper around a PlanExecutor rather than acting as a query execution operator in its own right.

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: