[SERVER-14961] Plan ranker favors intersection plans if predicate generates empty range index scan Created: 19/Aug/14  Updated: 11/Jul/16  Resolved: 25/Aug/14

Status: Closed
Project: Core Server
Component/s: Querying
Affects Version/s: 2.6.4
Fix Version/s: 2.6.5, 2.7.6

Type: Bug Priority: Major - P3
Reporter: Nic Cottrell (Personal) Assignee: J Rassi
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive mongod.zip    
Issue Links:
Depends
Duplicate
is duplicated by SERVER-15049 Database is read-overloaded after upg... Closed
is duplicated by SERVER-15196 Hashed index causes 'Overflow hashed ... Closed
is duplicated by SERVER-15300 Wrong index is being used, causing in... Closed
Related
related to SERVER-14311 skipping of index keys is not account... Closed
related to SERVER-15279 Disable hash-based index intersection... Closed
Tested
Operating System: ALL
Backport Completed:
Participants:

 Description   
Issue Status as of Aug 21, 2014

ISSUE SUMMARY
A change was introduced to the query engine in version 2.6.4 of the server, which affected the behavior of the query plan ranker. This change causes index intersection plans to be incorrectly chosen instead of a single-index plans, for certain types of queries with eligible index intersection plans. The incorrect choice of an index intersection plan in these cases introduces a performance regression to the affected queries.

A query is affected by this issue if it meets both of the following conditions:

  • the query includes a logical AND condition over multiple indexed predicates (i.e. the query is eligible for index intersection)
  • one of the indexed predicates in the AND condition matches no documents

Consider the following shell snippet:

db.foo.drop()
db.foo.ensureIndex({a:1})
db.foo.ensureIndex({b:1})
db.foo.insert({a:1,b:1})
db.foo.find({a:0,b:{$gte:1}})

The query on line 5 above generates an index intersection plan, and no documents from the collection match the indexed predicate {a: 0}. Thus, the query is affected by this issue.

USER IMPACT
Any query that is incorrectly assigned index intersection plans in this way will return correct results, but may suffer from reduced performance. In addition, the intersection plan will be saved in the query plan cache, such that subsequently executions of a query with the same shape may also suffer from reduced performance.

WORKAROUNDS
If the use of the intersection plan adversely affects performance, users may disable the query planner from generating intersection plans altogether by running the following from the shell:

db.adminCommand({setParameter: 1, internalQueryPlannerEnableIndexIntersection: 0});

Alternatively, users can restart mongod with option --setParameter internalQueryPlannerEnableIndexIntersection=0 to achieve the same effect.

AFFECTED VERSIONS
Only the MongoDB 2.6.4 production release is affected by this issue.

FIX VERSION
The fix is included in the 2.6.5 production release.

RESOLUTION DETAILS
The IndexScan query execution stage was changed such that single index plans are now correctly ranked higher than index intersection plans for the affected queries.

Original description

After upgrading to 2.6.4 the query optimizer is chosing crazy indexes (or combination of). For example, this query:

sprawk2:PRIMARY> db.Example.find( { clExists: true, lc: "eng", trans: "ara", tLcLeft: "conference", textLc: "conference", pattern: false, group: "all" }).explain();
{
	"cursor" : "Complex Plan",
	"n" : 0,
	"nscannedObjects" : 0,
	"nscanned" : 2,
	"nscannedObjectsAllPlans" : 0,
	"nscannedAllPlans" : 11,
	"nYields" : 4,
	"nChunkSkips" : 0,
	"millis" : 17320,
	"server" : "tor:27018",
	"filterSet" : false
}

I have an index

{trans:1,tLcLeft:1,group:1,lc:1}

also a sparse index on

{clExists:1,tLcLeft:1}

. Since about 1% of documents have a clExists field, this is pretty small, but either of these indices alone should give a quick response I would think. What is this complex query and why is it so slow?
Our whole cluster is overloading with these queries.



 Comments   
Comment by Githook User [ 25/Aug/14 ]

Author:

{u'username': u'jrassi', u'name': u'Jason Rassi', u'email': u'rassi@10gen.com'}

Message: SERVER-14961 IndexScan over empty range should return EOF immediately

(cherry picked from commit 2c510c5c06b4e557efb680d57694d24f1c028464)
Branch: v2.6
https://github.com/mongodb/mongo/commit/06e03e2201b0b2a46a277e52a674fd5d6f861c9e

Comment by Githook User [ 25/Aug/14 ]

Author:

{u'username': u'jrassi', u'name': u'Jason Rassi', u'email': u'rassi@10gen.com'}

Message: SERVER-14961 IndexScan over empty range should return EOF immediately
Branch: master
https://github.com/mongodb/mongo/commit/2c510c5c06b4e557efb680d57694d24f1c028464

Comment by J Rassi [ 24/Aug/14 ]

You'll have to enable it separately on each 2.6.4 mongod in your cluster.

Comment by Nic Cottrell (Personal) [ 24/Aug/14 ]

Thanks Jason. Do I need to run that on each node on the cluster? Or is there a way of propagating it?

Comment by J Rassi [ 22/Aug/14 ]

niccottrell, run the following command at the shell to stop the query planner from generating intersection plans:

db.adminCommand({setParameter: 1, internalQueryPlannerEnableIndexIntersection: 0});

Alternatively, restart the server with option "--setParameter internalQueryPlannerEnableIndexIntersection=0".

Comment by Nic Cottrell (Personal) [ 22/Aug/14 ]

@Jason - thanks for the update. Anything on workarounds yet? I've added some new indices which helps a little. Should I remove the hashed ones on text and textLc?

Comment by J Rassi [ 21/Aug/14 ]

Thanks, Nic. I read the output you provided and investigated further, and believe I have found the root cause of the issue.

In version 2.6.4 of the server the query planner will choose (and cache) index intersection plans for queries that return zero results, in situations where it is expected that a single-index plan would be chosen. I can reproduce this issue as follows:

> db.foo.drop()
true
> db.foo.insert({a:1,b:1})
> db.foo.ensureIndex({a:1})
> db.foo.ensureIndex({b:1})
> db.foo.find({a:0,b:{$gte:1}}).explain(true)
{
	"cursor" : "Complex Plan",
	"n" : 0,
	"nscannedObjects" : 0,
	"nscanned" : 1,
	"nscannedObjectsAllPlans" : 0,
	"nscannedAllPlans" : 1,
	"nYields" : 0,
	"nChunkSkips" : 0,
	"millis" : 0,
	"allPlans" : [
		{
			"cursor" : "Complex Plan",
			"n" : 0,
			"nscannedObjects" : 0,
			"nscanned" : 1,
			"nChunkSkips" : 0
		},
		{
			"cursor" : "BtreeCursor a_1",
			"isMultiKey" : false,
			"n" : 0,
			"nscannedObjects" : 0,
			"nscanned" : 0,
			"scanAndOrder" : false,
			"indexOnly" : false,
			"nChunkSkips" : 0,
			"indexBounds" : {
				"a" : [
					[
						0,
						0
					]
				]
			}
		}
	],
	"server" : "Rassi-MacBook-Pro.local:27017",
	"filterSet" : false,
	"stats" : {
		"type" : "FETCH",
		"works" : 2,
		"yields" : 0,
		"unyields" : 0,
		"invalidates" : 0,
		"advanced" : 0,
		"needTime" : 0,
		"needFetch" : 0,
		"isEOF" : 1,
		"alreadyHasObj" : 0,
		"forcedFetches" : 0,
		"matchTested" : 0,
		"children" : [
			{
				"type" : "KEEP_MUTATIONS",
				"works" : 1,
				"yields" : 0,
				"unyields" : 0,
				"invalidates" : 0,
				"advanced" : 0,
				"needTime" : 0,
				"needFetch" : 0,
				"isEOF" : 1,
				"children" : [
					{
						"type" : "AND_HASH",
						"works" : 1,
						"yields" : 0,
						"unyields" : 0,
						"invalidates" : 0,
						"advanced" : 0,
						"needTime" : 0,
						"needFetch" : 0,
						"isEOF" : 1,
						"flaggedButPassed" : 0,
						"flaggedInProgress" : 0,
						"memUsage" : 0,
						"memLimit" : 33554432,
						"children" : [
							{
								"type" : "IXSCAN",
								"works" : 2,
								"yields" : 0,
								"unyields" : 0,
								"invalidates" : 0,
								"advanced" : 1,
								"needTime" : 1,
								"needFetch" : 0,
								"isEOF" : 1,
								"keyPattern" : "{ b: 1.0 }",
								"isMultiKey" : 0,
								"boundsVerbose" : "field #0['b']: [1.0, inf.0]",
								"yieldMovedCursor" : 0,
								"dupsTested" : 0,
								"dupsDropped" : 0,
								"seenInvalidated" : 0,
								"matchTested" : 0,
								"keysExamined" : 1,
								"children" : [ ]
							},
							{
								"type" : "IXSCAN",
								"works" : 2,
								"yields" : 0,
								"unyields" : 0,
								"invalidates" : 0,
								"advanced" : 0,
								"needTime" : 1,
								"needFetch" : 0,
								"isEOF" : 1,
								"keyPattern" : "{ a: 1.0 }",
								"isMultiKey" : 0,
								"boundsVerbose" : "field #0['a']: [0.0, 0.0]",
								"yieldMovedCursor" : 0,
								"dupsTested" : 0,
								"dupsDropped" : 0,
								"seenInvalidated" : 0,
								"matchTested" : 0,
								"keysExamined" : 0,
								"children" : [ ]
							}
						]
					}
				]
			}
		]
	}
}

Because no documents match the {a: 0} predicate, the {a: 1} index is the optimal index for the query on this collection. However, the query planner is instead choosing a plan that intersects results from the {a: 1} index with the {b: 1} index. In this specific case where the intersection plan returns no results because one of the indexed predicates matches no documents, the intersection plan will perform about the same as the single-index plan. However, since the query planner caches the winning plan for a shape after its first execution, subsequent queries of this shape (depending on the data distribution) will use the intersection plan, which may not perform as well as the single-index plan.

This patch for SERVER-14311 (which was included in the 2.6.4 release) contains the root cause of this issue. It modifies the index scan query stage such that two "work cycles" are needed to complete an index scan over an empty range, whereas only one "work cycle" was needed in 2.6.3. When multiple query plans are available, the plan ranker chooses one by seeing which takes the fewest number of work cycles to complete the query (or which made the most progress along the way, if no plan has finished after a large number of work cycles). For the query shown above, the plan that chooses a single index plan completes in two work cycles, whereas the index intersection plan completes in only one work cycle. Thus, the intersection plan wins. The reason the intersection plan only takes one work cycle to complete is because the hashed intersection query stage contains special "lookahead" logic that runs many work cycles on the underlying index scan stages to detect early completion of the index scan. This logic was included as part of a pass intending to tweak the plan ranking of intersection plans.

The output you posted shows a manifestation of this issue for your query. It includes predicates on the "textLc" and "group" fields, and no documents match the predicate on "textLc". Similar to my example above, it is expected that a single-index plan is chosen using the {textLc: "hashed"} index, but instead an index intersection plan is chosen that uses both the {textLc: "hashed"} index and the {group: 1, lastModified: 1} index.

Thanks for reporting the issue. Please continue to watch this ticket for updates on the fix schedule, and for possible workaround information.

~ Jason Rassi

Comment by Nic Cottrell (Personal) [ 20/Aug/14 ]

@Jason - thanks and here are the results... (Since there are no matches on itcount(), the

{clExists:1,tLcLeft:1}

index would give zero matches, so strange that it would need to intersect anything - could stop right there)...

sprawk2:PRIMARY> var q = {clExists: true, lc: "eng", trans: "ara", tLcLeft: "conference", textLc: "conference", pattern: false, group: "all"};
sprawk2:PRIMARY> db.Example.find(q).explain(true);
{
	"cursor" : "Complex Plan",
	"n" : 0,
	"nscannedObjects" : 0,
	"nscanned" : 2,
	"nscannedObjectsAllPlans" : 0,
	"nscannedAllPlans" : 11,
	"nYields" : 2,
	"nChunkSkips" : 0,
	"millis" : 229,
	"allPlans" : [
		{
			"cursor" : "Complex Plan",
			"n" : 0,
			"nscannedObjects" : 0,
			"nscanned" : 2,
			"nChunkSkips" : 0
		},
		{
			"cursor" : "Complex Plan",
			"n" : 0,
			"nscannedObjects" : 0,
			"nscanned" : 2,
			"nChunkSkips" : 0
		},
		{
			"cursor" : "BtreeCursor lc_1_group_1_tLcLeft_1",
			"isMultiKey" : false,
			"n" : 0,
			"nscannedObjects" : 0,
			"nscanned" : 0,
			"scanAndOrder" : false,
			"indexOnly" : false,
			"nChunkSkips" : 0,
			"indexBounds" : {
				"lc" : [
					[
						"eng",
						"eng"
					]
				],
				"group" : [
					[
						"all",
						"all"
					]
				],
				"tLcLeft" : [
					[
						"conference",
						"conference"
					]
				]
			}
		},
		{
			"cursor" : "BtreeCursor tLcLeft_1_group_1_trans_1",
			"isMultiKey" : true,
			"n" : 0,
			"nscannedObjects" : 0,
			"nscanned" : 0,
			"scanAndOrder" : false,
			"indexOnly" : false,
			"nChunkSkips" : 0,
			"indexBounds" : {
				"tLcLeft" : [
					[
						"conference",
						"conference"
					]
				],
				"group" : [
					[
						"all",
						"all"
					]
				],
				"trans" : [
					[
						"ara",
						"ara"
					]
				]
			}
		},
		{
			"cursor" : "BtreeCursor lc_1_group_1_trans_1_wordCount_1_pattern_1",
			"isMultiKey" : true,
			"n" : 0,
			"nscannedObjects" : 0,
			"nscanned" : 1,
			"scanAndOrder" : false,
			"indexOnly" : false,
			"nChunkSkips" : 0,
			"indexBounds" : {
				"lc" : [
					[
						"eng",
						"eng"
					]
				],
				"group" : [
					[
						"all",
						"all"
					]
				],
				"trans" : [
					[
						"ara",
						"ara"
					]
				],
				"wordCount" : [
					[
						{
							"$minElement" : 1
						},
						{
							"$maxElement" : 1
						}
					]
				],
				"pattern" : [
					[
						false,
						false
					]
				]
			}
		},
		{
			"cursor" : "BtreeCursor group_1_pass_1",
			"isMultiKey" : false,
			"n" : 0,
			"nscannedObjects" : 0,
			"nscanned" : 1,
			"scanAndOrder" : false,
			"indexOnly" : false,
			"nChunkSkips" : 0,
			"indexBounds" : {
				"group" : [
					[
						"all",
						"all"
					]
				],
				"pass" : [
					[
						{
							"$minElement" : 1
						},
						{
							"$maxElement" : 1
						}
					]
				]
			}
		},
		{
			"cursor" : "BtreeCursor clExists_1_tLcLeft_1",
			"isMultiKey" : false,
			"n" : 0,
			"nscannedObjects" : 0,
			"nscanned" : 0,
			"scanAndOrder" : false,
			"indexOnly" : false,
			"nChunkSkips" : 0,
			"indexBounds" : {
				"clExists" : [
					[
						true,
						true
					]
				],
				"tLcLeft" : [
					[
						"conference",
						"conference"
					]
				]
			}
		},
		{
			"cursor" : "BtreeCursor trans_1_tLcLeft_1_group_1_lc_1",
			"isMultiKey" : true,
			"n" : 0,
			"nscannedObjects" : 0,
			"nscanned" : 0,
			"scanAndOrder" : false,
			"indexOnly" : false,
			"nChunkSkips" : 0,
			"indexBounds" : {
				"trans" : [
					[
						"ara",
						"ara"
					]
				],
				"tLcLeft" : [
					[
						"conference",
						"conference"
					]
				],
				"group" : [
					[
						"all",
						"all"
					]
				],
				"lc" : [
					[
						"eng",
						"eng"
					]
				]
			}
		},
		{
			"cursor" : "BtreeCursor textLc_hashed",
			"isMultiKey" : false,
			"n" : 0,
			"nscannedObjects" : 0,
			"nscanned" : 1,
			"scanAndOrder" : false,
			"indexOnly" : false,
			"nChunkSkips" : 0,
			"indexBounds" : {
				"textLc" : [
					[
						NumberLong("6337680983347079397"),
						NumberLong("6337680983347079397")
					]
				]
			}
		},
		{
			"cursor" : "Complex Plan",
			"n" : 0,
			"nscannedObjects" : 0,
			"nscanned" : 4,
			"nChunkSkips" : 0
		},
		{
			"cursor" : "Complex Plan",
			"n" : 0,
			"nscannedObjects" : 0,
			"nscanned" : 0,
			"nChunkSkips" : 0
		}
	],
	"server" : "tor:27018",
	"filterSet" : false,
	"stats" : {
		"type" : "KEEP_MUTATIONS",
		"works" : 2,
		"yields" : 2,
		"unyields" : 2,
		"invalidates" : 0,
		"advanced" : 0,
		"needTime" : 0,
		"needFetch" : 0,
		"isEOF" : 1,
		"children" : [
			{
				"type" : "FETCH",
				"works" : 1,
				"yields" : 2,
				"unyields" : 2,
				"invalidates" : 0,
				"advanced" : 0,
				"needTime" : 0,
				"needFetch" : 0,
				"isEOF" : 1,
				"alreadyHasObj" : 0,
				"forcedFetches" : 0,
				"matchTested" : 0,
				"children" : [
					{
						"type" : "AND_HASH",
						"works" : 1,
						"yields" : 2,
						"unyields" : 2,
						"invalidates" : 0,
						"advanced" : 0,
						"needTime" : 0,
						"needFetch" : 0,
						"isEOF" : 1,
						"flaggedButPassed" : 0,
						"flaggedInProgress" : 0,
						"memUsage" : 0,
						"memLimit" : 33554432,
						"children" : [
							{
								"type" : "IXSCAN",
								"works" : 2,
								"yields" : 2,
								"unyields" : 2,
								"invalidates" : 0,
								"advanced" : 1,
								"needTime" : 1,
								"needFetch" : 0,
								"isEOF" : 0,
								"keyPattern" : "{ group: 1.0, lastModified: 1.0 }",
								"isMultiKey" : 0,
								"boundsVerbose" : "field #0['group']: [\"all\", \"all\"], field #1['lastModified']: [MinKey, MaxKey]",
								"yieldMovedCursor" : 0,
								"dupsTested" : 0,
								"dupsDropped" : 0,
								"seenInvalidated" : 0,
								"matchTested" : 0,
								"keysExamined" : 2,
								"children" : [ ]
							},
							{
								"type" : "IXSCAN",
								"works" : 2,
								"yields" : 2,
								"unyields" : 2,
								"invalidates" : 0,
								"advanced" : 0,
								"needTime" : 1,
								"needFetch" : 0,
								"isEOF" : 1,
								"keyPattern" : "{ clExists: 1.0, tLcLeft: 1.0 }",
								"isMultiKey" : 0,
								"boundsVerbose" : "field #0['clExists']: [true, true], field #1['tLcLeft']: [\"conference\", \"conference\"]",
								"yieldMovedCursor" : 0,
								"dupsTested" : 0,
								"dupsDropped" : 0,
								"seenInvalidated" : 0,
								"matchTested" : 0,
								"keysExamined" : 0,
								"children" : [ ]
							}
						]
					}
				]
			}
		]
	}
}
sprawk2:PRIMARY> db.Example.find(q).itcount();
 
0
sprawk2:PRIMARY> db.Example.getPlanCache().getPlansByQuery(q);
[
	{
		"details" : {
			"solution" : "(index-tagged expression tree: tree=Node\n---Leaf \n---Leaf { group: 1.0, lastModified: 1.0 }, pos: 0\n---Leaf \n---Leaf \n---Leaf \n---Leaf { textLc: \"hashed\" }, pos: 0\n---Leaf \n)"
		},
		"reason" : {
			"score" : 1.0002,
			"stats" : {
				"type" : "LIMIT",
				"works" : 1,
				"yields" : 1,
				"unyields" : 1,
				"invalidates" : 0,
				"advanced" : 0,
				"needTime" : 0,
				"needFetch" : 0,
				"isEOF" : 1,
				"children" : [
					{
						"type" : "KEEP_MUTATIONS",
						"works" : 1,
						"yields" : 1,
						"unyields" : 1,
						"invalidates" : 0,
						"advanced" : 0,
						"needTime" : 0,
						"needFetch" : 0,
						"isEOF" : 1,
						"children" : [
							{
								"type" : "FETCH",
								"works" : 1,
								"yields" : 1,
								"unyields" : 1,
								"invalidates" : 0,
								"advanced" : 0,
								"needTime" : 0,
								"needFetch" : 0,
								"isEOF" : 1,
								"alreadyHasObj" : 0,
								"forcedFetches" : 0,
								"matchTested" : 0,
								"children" : [
									{
										"type" : "AND_HASH",
										"works" : 1,
										"yields" : 1,
										"unyields" : 1,
										"invalidates" : 0,
										"advanced" : 0,
										"needTime" : 0,
										"needFetch" : 0,
										"isEOF" : 1,
										"flaggedButPassed" : 0,
										"flaggedInProgress" : 0,
										"memUsage" : 0,
										"memLimit" : 33554432,
										"children" : [
											{
												"type" : "IXSCAN",
												"works" : 2,
												"yields" : 1,
												"unyields" : 1,
												"invalidates" : 0,
												"advanced" : 1,
												"needTime" : 1,
												"needFetch" : 0,
												"isEOF" : 0,
												"keyPattern" : "{ group: 1.0, lastModified: 1.0 }",
												"isMultiKey" : 0,
												"boundsVerbose" : "field #0['group']: [\"cn\", \"cn\"], field #1['lastModified']: [MinKey, MaxKey]",
												"yieldMovedCursor" : 0,
												"dupsTested" : 0,
												"dupsDropped" : 0,
												"seenInvalidated" : 0,
												"matchTested" : 0,
												"keysExamined" : 2,
												"children" : [ ]
											},
											{
												"type" : "IXSCAN",
												"works" : 2,
												"yields" : 1,
												"unyields" : 1,
												"invalidates" : 0,
												"advanced" : 0,
												"needTime" : 1,
												"needFetch" : 0,
												"isEOF" : 1,
												"keyPattern" : "{ textLc: \"hashed\" }",
												"isMultiKey" : 0,
												"boundsVerbose" : "field #0['textLc']: [-3202514460983226863, -3202514460983226863]",
												"yieldMovedCursor" : 0,
												"dupsTested" : 0,
												"dupsDropped" : 0,
												"seenInvalidated" : 0,
												"matchTested" : 0,
												"keysExamined" : 0,
												"children" : [ ]
											}
										]
									}
								]
							}
						]
					}
				]
			}
		},
		"feedback" : {
			"nfeedback" : 20,
			"averageScore" : 1.0001999999999998,
			"stdDevScore" : 2.278129578503827e-16,
			"scores" : [
				{
					"score" : 1.0002
				},
				{
					"score" : 1.0002
				},
				{
					"score" : 1.0002
				},
				{
					"score" : 1.0002
				},
				{
					"score" : 1.0002
				},
				{
					"score" : 1.0002
				},
				{
					"score" : 1.0002
				},
				{
					"score" : 1.0002
				},
				{
					"score" : 1.0002
				},
				{
					"score" : 1.0002
				},
				{
					"score" : 1.0002
				},
				{
					"score" : 1.0002
				},
				{
					"score" : 1.0002
				},
				{
					"score" : 1.0002
				},
				{
					"score" : 1.0002
				},
				{
					"score" : 1.0002
				},
				{
					"score" : 1.0002
				},
				{
					"score" : 1.0002
				},
				{
					"score" : 1.0002
				},
				{
					"score" : 1.0002
				}
			]
		},
		"filterSet" : false
	},
	{
		"details" : {
			"solution" : "(index-tagged expression tree: tree=Node\n---Leaf \n---Leaf { group: 1.0, lastModified: 1.0 }, pos: 0\n---Leaf \n---Leaf \n---Leaf \n---Leaf \n---Leaf \n)"
		},
		"reason" : {
			"score" : 1.0003,
			"stats" : {
				"type" : "LIMIT",
				"works" : 1,
				"yields" : 1,
				"unyields" : 1,
				"invalidates" : 0,
				"advanced" : 0,
				"needTime" : 1,
				"needFetch" : 0,
				"isEOF" : 0,
				"children" : [
					{
						"type" : "KEEP_MUTATIONS",
						"works" : 1,
						"yields" : 1,
						"unyields" : 1,
						"invalidates" : 0,
						"advanced" : 0,
						"needTime" : 1,
						"needFetch" : 0,
						"isEOF" : 0,
						"children" : [
							{
								"type" : "FETCH",
								"works" : 1,
								"yields" : 1,
								"unyields" : 1,
								"invalidates" : 0,
								"advanced" : 0,
								"needTime" : 1,
								"needFetch" : 0,
								"isEOF" : 0,
								"alreadyHasObj" : 0,
								"forcedFetches" : 0,
								"matchTested" : 0,
								"children" : [
									{
										"type" : "IXSCAN",
										"works" : 1,
										"yields" : 1,
										"unyields" : 1,
										"invalidates" : 0,
										"advanced" : 0,
										"needTime" : 1,
										"needFetch" : 0,
										"isEOF" : 0,
										"keyPattern" : "{ group: 1.0, lastModified: 1.0 }",
										"isMultiKey" : 0,
										"boundsVerbose" : "field #0['group']: [\"cn\", \"cn\"], field #1['lastModified']: [MinKey, MaxKey]",
										"yieldMovedCursor" : 0,
										"dupsTested" : 0,
										"dupsDropped" : 0,
										"seenInvalidated" : 0,
										"matchTested" : 0,
										"keysExamined" : 1,
										"children" : [ ]
									}
								]
							}
						]
					}
				]
			}
		},
		"feedback" : {
			
		},
		"filterSet" : false
	},
	{
		"details" : {
			"solution" : "(index-tagged expression tree: tree=Node\n---Leaf \n---Leaf { lc: 1.0, group: 1.0, tLcLeft: 1.0 }, pos: 1\n---Leaf { lc: 1.0, group: 1.0, tLcLeft: 1.0 }, pos: 0\n---Leaf \n---Leaf { lc: 1.0, group: 1.0, tLcLeft: 1.0 }, pos: 2\n---Leaf \n---Leaf \n)"
		},
		"reason" : {
			"score" : 1.0003,
			"stats" : {
				"type" : "LIMIT",
				"works" : 1,
				"yields" : 1,
				"unyields" : 1,
				"invalidates" : 0,
				"advanced" : 0,
				"needTime" : 1,
				"needFetch" : 0,
				"isEOF" : 0,
				"children" : [
					{
						"type" : "KEEP_MUTATIONS",
						"works" : 1,
						"yields" : 1,
						"unyields" : 1,
						"invalidates" : 0,
						"advanced" : 0,
						"needTime" : 1,
						"needFetch" : 0,
						"isEOF" : 0,
						"children" : [
							{
								"type" : "FETCH",
								"works" : 1,
								"yields" : 1,
								"unyields" : 1,
								"invalidates" : 0,
								"advanced" : 0,
								"needTime" : 1,
								"needFetch" : 0,
								"isEOF" : 0,
								"alreadyHasObj" : 0,
								"forcedFetches" : 0,
								"matchTested" : 0,
								"children" : [
									{
										"type" : "IXSCAN",
										"works" : 1,
										"yields" : 1,
										"unyields" : 1,
										"invalidates" : 0,
										"advanced" : 0,
										"needTime" : 1,
										"needFetch" : 0,
										"isEOF" : 0,
										"keyPattern" : "{ lc: 1.0, group: 1.0, tLcLeft: 1.0 }",
										"isMultiKey" : 0,
										"boundsVerbose" : "field #0['lc']: [\"eng\", \"eng\"], field #1['group']: [\"cn\", \"cn\"], field #2['tLcLeft']: [\"the national ass\", \"the national ass\"]",
										"yieldMovedCursor" : 0,
										"dupsTested" : 0,
										"dupsDropped" : 0,
										"seenInvalidated" : 0,
										"matchTested" : 0,
										"keysExamined" : 1,
										"children" : [ ]
									}
								]
							}
						]
					}
				]
			}
		},
		"feedback" : {
			
		},
		"filterSet" : false
	},
	{
		"details" : {
			"solution" : "(index-tagged expression tree: tree=Node\n---Leaf \n---Leaf { tLcLeft: 1, group: 1, trans: 1 }, pos: 1\n---Leaf \n---Leaf \n---Leaf { tLcLeft: 1, group: 1, trans: 1 }, pos: 0\n---Leaf \n---Leaf { tLcLeft: 1, group: 1, trans: 1 }, pos: 2\n)"
		},
		"reason" : {
			"score" : 1.0003,
			"stats" : {
				"type" : "LIMIT",
				"works" : 1,
				"yields" : 1,
				"unyields" : 1,
				"invalidates" : 0,
				"advanced" : 0,
				"needTime" : 1,
				"needFetch" : 0,
				"isEOF" : 0,
				"children" : [
					{
						"type" : "KEEP_MUTATIONS",
						"works" : 1,
						"yields" : 1,
						"unyields" : 1,
						"invalidates" : 0,
						"advanced" : 0,
						"needTime" : 1,
						"needFetch" : 0,
						"isEOF" : 0,
						"children" : [
							{
								"type" : "FETCH",
								"works" : 1,
								"yields" : 1,
								"unyields" : 1,
								"invalidates" : 0,
								"advanced" : 0,
								"needTime" : 1,
								"needFetch" : 0,
								"isEOF" : 0,
								"alreadyHasObj" : 0,
								"forcedFetches" : 0,
								"matchTested" : 0,
								"children" : [
									{
										"type" : "IXSCAN",
										"works" : 1,
										"yields" : 1,
										"unyields" : 1,
										"invalidates" : 0,
										"advanced" : 0,
										"needTime" : 1,
										"needFetch" : 0,
										"isEOF" : 0,
										"keyPattern" : "{ tLcLeft: 1, group: 1, trans: 1 }",
										"isMultiKey" : 1,
										"boundsVerbose" : "field #0['tLcLeft']: [\"the national ass\", \"the national ass\"], field #1['group']: [\"cn\", \"cn\"], field #2['trans']: [\"esl\", \"esl\"]",
										"yieldMovedCursor" : 0,
										"dupsTested" : 0,
										"dupsDropped" : 0,
										"seenInvalidated" : 0,
										"matchTested" : 0,
										"keysExamined" : 1,
										"children" : [ ]
									}
								]
							}
						]
					}
				]
			}
		},
		"feedback" : {
			
		},
		"filterSet" : false
	},
	{
		"details" : {
			"solution" : "(index-tagged expression tree: tree=Node\n---Leaf \n---Leaf { lc: 1.0, group: 1.0, trans: 1.0, wordCount: 1.0, pattern: 1.0 }, pos: 1\n---Leaf { lc: 1.0, group: 1.0, trans: 1.0, wordCount: 1.0, pattern: 1.0 }, pos: 0\n---Leaf { lc: 1.0, group: 1.0, trans: 1.0, wordCount: 1.0, pattern: 1.0 }, pos: 4\n---Leaf \n---Leaf \n---Leaf { lc: 1.0, group: 1.0, trans: 1.0, wordCount: 1.0, pattern: 1.0 }, pos: 2\n)"
		},
		"reason" : {
			"score" : 1.0003,
			"stats" : {
				"type" : "LIMIT",
				"works" : 1,
				"yields" : 1,
				"unyields" : 1,
				"invalidates" : 0,
				"advanced" : 0,
				"needTime" : 1,
				"needFetch" : 0,
				"isEOF" : 0,
				"children" : [
					{
						"type" : "KEEP_MUTATIONS",
						"works" : 1,
						"yields" : 1,
						"unyields" : 1,
						"invalidates" : 0,
						"advanced" : 0,
						"needTime" : 1,
						"needFetch" : 0,
						"isEOF" : 0,
						"children" : [
							{
								"type" : "FETCH",
								"works" : 1,
								"yields" : 1,
								"unyields" : 1,
								"invalidates" : 0,
								"advanced" : 0,
								"needTime" : 1,
								"needFetch" : 0,
								"isEOF" : 0,
								"alreadyHasObj" : 0,
								"forcedFetches" : 0,
								"matchTested" : 0,
								"children" : [
									{
										"type" : "IXSCAN",
										"works" : 1,
										"yields" : 1,
										"unyields" : 1,
										"invalidates" : 0,
										"advanced" : 0,
										"needTime" : 1,
										"needFetch" : 0,
										"isEOF" : 0,
										"keyPattern" : "{ lc: 1.0, group: 1.0, trans: 1.0, wordCount: 1.0, pattern: 1.0 }",
										"isMultiKey" : 1,
										"boundsVerbose" : "field #0['lc']: [\"eng\", \"eng\"], field #1['group']: [\"cn\", \"cn\"], field #2['trans']: [\"esl\", \"esl\"], field #3['wordCount']: [MinKey, MaxKey], field #4['pattern']: [false, false]",
										"yieldMovedCursor" : 0,
										"dupsTested" : 0,
										"dupsDropped" : 0,
										"seenInvalidated" : 0,
										"matchTested" : 0,
										"keysExamined" : 1,
										"children" : [ ]
									}
								]
							}
						]
					}
				]
			}
		},
		"feedback" : {
			
		},
		"filterSet" : false
	},
	{
		"details" : {
			"solution" : "(index-tagged expression tree: tree=Node\n---Leaf \n---Leaf { group: 1.0, pass: 1.0 }, pos: 0\n---Leaf \n---Leaf \n---Leaf \n---Leaf \n---Leaf \n)"
		},
		"reason" : {
			"score" : 1.0003,
			"stats" : {
				"type" : "LIMIT",
				"works" : 1,
				"yields" : 1,
				"unyields" : 1,
				"invalidates" : 0,
				"advanced" : 0,
				"needTime" : 1,
				"needFetch" : 0,
				"isEOF" : 0,
				"children" : [
					{
						"type" : "KEEP_MUTATIONS",
						"works" : 1,
						"yields" : 1,
						"unyields" : 1,
						"invalidates" : 0,
						"advanced" : 0,
						"needTime" : 1,
						"needFetch" : 0,
						"isEOF" : 0,
						"children" : [
							{
								"type" : "FETCH",
								"works" : 1,
								"yields" : 1,
								"unyields" : 1,
								"invalidates" : 0,
								"advanced" : 0,
								"needTime" : 1,
								"needFetch" : 0,
								"isEOF" : 0,
								"alreadyHasObj" : 0,
								"forcedFetches" : 0,
								"matchTested" : 0,
								"children" : [
									{
										"type" : "IXSCAN",
										"works" : 1,
										"yields" : 1,
										"unyields" : 1,
										"invalidates" : 0,
										"advanced" : 0,
										"needTime" : 1,
										"needFetch" : 0,
										"isEOF" : 0,
										"keyPattern" : "{ group: 1.0, pass: 1.0 }",
										"isMultiKey" : 0,
										"boundsVerbose" : "field #0['group']: [\"cn\", \"cn\"], field #1['pass']: [MinKey, MaxKey]",
										"yieldMovedCursor" : 0,
										"dupsTested" : 0,
										"dupsDropped" : 0,
										"seenInvalidated" : 0,
										"matchTested" : 0,
										"keysExamined" : 1,
										"children" : [ ]
									}
								]
							}
						]
					}
				]
			}
		},
		"feedback" : {
			
		},
		"filterSet" : false
	},
	{
		"details" : {
			"solution" : "(index-tagged expression tree: tree=Node\n---Leaf { clExists: 1.0, tLcLeft: 1.0 }, pos: 0\n---Leaf \n---Leaf \n---Leaf \n---Leaf { clExists: 1.0, tLcLeft: 1.0 }, pos: 1\n---Leaf \n---Leaf \n)"
		},
		"reason" : {
			"score" : 1.0003,
			"stats" : {
				"type" : "LIMIT",
				"works" : 1,
				"yields" : 1,
				"unyields" : 1,
				"invalidates" : 0,
				"advanced" : 0,
				"needTime" : 1,
				"needFetch" : 0,
				"isEOF" : 0,
				"children" : [
					{
						"type" : "KEEP_MUTATIONS",
						"works" : 1,
						"yields" : 1,
						"unyields" : 1,
						"invalidates" : 0,
						"advanced" : 0,
						"needTime" : 1,
						"needFetch" : 0,
						"isEOF" : 0,
						"children" : [
							{
								"type" : "FETCH",
								"works" : 1,
								"yields" : 1,
								"unyields" : 1,
								"invalidates" : 0,
								"advanced" : 0,
								"needTime" : 1,
								"needFetch" : 0,
								"isEOF" : 0,
								"alreadyHasObj" : 0,
								"forcedFetches" : 0,
								"matchTested" : 0,
								"children" : [
									{
										"type" : "IXSCAN",
										"works" : 1,
										"yields" : 1,
										"unyields" : 1,
										"invalidates" : 0,
										"advanced" : 0,
										"needTime" : 1,
										"needFetch" : 0,
										"isEOF" : 0,
										"keyPattern" : "{ clExists: 1.0, tLcLeft: 1.0 }",
										"isMultiKey" : 0,
										"boundsVerbose" : "field #0['clExists']: [true, true], field #1['tLcLeft']: [\"the national ass\", \"the national ass\"]",
										"yieldMovedCursor" : 0,
										"dupsTested" : 0,
										"dupsDropped" : 0,
										"seenInvalidated" : 0,
										"matchTested" : 0,
										"keysExamined" : 1,
										"children" : [ ]
									}
								]
							}
						]
					}
				]
			}
		},
		"feedback" : {
			
		},
		"filterSet" : false
	},
	{
		"details" : {
			"solution" : "(index-tagged expression tree: tree=Node\n---Leaf \n---Leaf { trans: 1.0, tLcLeft: 1.0, group: 1.0, lc: 1.0 }, pos: 2\n---Leaf { trans: 1.0, tLcLeft: 1.0, group: 1.0, lc: 1.0 }, pos: 3\n---Leaf \n---Leaf { trans: 1.0, tLcLeft: 1.0, group: 1.0, lc: 1.0 }, pos: 1\n---Leaf \n---Leaf { trans: 1.0, tLcLeft: 1.0, group: 1.0, lc: 1.0 }, pos: 0\n)"
		},
		"reason" : {
			"score" : 1.0003,
			"stats" : {
				"type" : "LIMIT",
				"works" : 1,
				"yields" : 1,
				"unyields" : 1,
				"invalidates" : 0,
				"advanced" : 0,
				"needTime" : 1,
				"needFetch" : 0,
				"isEOF" : 0,
				"children" : [
					{
						"type" : "KEEP_MUTATIONS",
						"works" : 1,
						"yields" : 1,
						"unyields" : 1,
						"invalidates" : 0,
						"advanced" : 0,
						"needTime" : 1,
						"needFetch" : 0,
						"isEOF" : 0,
						"children" : [
							{
								"type" : "FETCH",
								"works" : 1,
								"yields" : 1,
								"unyields" : 1,
								"invalidates" : 0,
								"advanced" : 0,
								"needTime" : 1,
								"needFetch" : 0,
								"isEOF" : 0,
								"alreadyHasObj" : 0,
								"forcedFetches" : 0,
								"matchTested" : 0,
								"children" : [
									{
										"type" : "IXSCAN",
										"works" : 1,
										"yields" : 1,
										"unyields" : 1,
										"invalidates" : 0,
										"advanced" : 0,
										"needTime" : 1,
										"needFetch" : 0,
										"isEOF" : 0,
										"keyPattern" : "{ trans: 1.0, tLcLeft: 1.0, group: 1.0, lc: 1.0 }",
										"isMultiKey" : 1,
										"boundsVerbose" : "field #0['trans']: [\"esl\", \"esl\"], field #1['tLcLeft']: [\"the national ass\", \"the national ass\"], field #2['group']: [\"cn\", \"cn\"], field #3['lc']: [\"eng\", \"eng\"]",
										"yieldMovedCursor" : 0,
										"dupsTested" : 0,
										"dupsDropped" : 0,
										"seenInvalidated" : 0,
										"matchTested" : 0,
										"keysExamined" : 1,
										"children" : [ ]
									}
								]
							}
						]
					}
				]
			}
		},
		"feedback" : {
			
		},
		"filterSet" : false
	},
	{
		"details" : {
			"solution" : "(index-tagged expression tree: tree=Node\n---Leaf \n---Leaf \n---Leaf \n---Leaf \n---Leaf \n---Leaf { textLc: \"hashed\" }, pos: 0\n---Leaf \n)"
		},
		"reason" : {
			"score" : 1.0003,
			"stats" : {
				"type" : "LIMIT",
				"works" : 1,
				"yields" : 1,
				"unyields" : 1,
				"invalidates" : 0,
				"advanced" : 0,
				"needTime" : 1,
				"needFetch" : 0,
				"isEOF" : 0,
				"children" : [
					{
						"type" : "KEEP_MUTATIONS",
						"works" : 1,
						"yields" : 1,
						"unyields" : 1,
						"invalidates" : 0,
						"advanced" : 0,
						"needTime" : 1,
						"needFetch" : 0,
						"isEOF" : 0,
						"children" : [
							{
								"type" : "FETCH",
								"works" : 1,
								"yields" : 1,
								"unyields" : 1,
								"invalidates" : 0,
								"advanced" : 0,
								"needTime" : 1,
								"needFetch" : 0,
								"isEOF" : 1,
								"alreadyHasObj" : 0,
								"forcedFetches" : 0,
								"matchTested" : 0,
								"children" : [
									{
										"type" : "IXSCAN",
										"works" : 1,
										"yields" : 1,
										"unyields" : 1,
										"invalidates" : 0,
										"advanced" : 0,
										"needTime" : 1,
										"needFetch" : 0,
										"isEOF" : 1,
										"keyPattern" : "{ textLc: \"hashed\" }",
										"isMultiKey" : 0,
										"boundsVerbose" : "field #0['textLc']: [-3202514460983226863, -3202514460983226863]",
										"yieldMovedCursor" : 0,
										"dupsTested" : 0,
										"dupsDropped" : 0,
										"seenInvalidated" : 0,
										"matchTested" : 0,
										"keysExamined" : 0,
										"children" : [ ]
									}
								]
							}
						]
					}
				]
			}
		},
		"feedback" : {
			
		},
		"filterSet" : false
	},
	{
		"details" : {
			"solution" : "(index-tagged expression tree: tree=Node\n---Leaf { clExists: 1.0, tLcLeft: 1.0 }, pos: 0\n---Leaf { group: 1.0, lastModified: 1.0 }, pos: 0\n---Leaf \n---Leaf \n---Leaf { clExists: 1.0, tLcLeft: 1.0 }, pos: 1\n---Leaf \n---Leaf \n)"
		},
		"reason" : {
			"score" : 1.0002,
			"stats" : {
				"type" : "LIMIT",
				"works" : 1,
				"yields" : 1,
				"unyields" : 1,
				"invalidates" : 0,
				"advanced" : 0,
				"needTime" : 1,
				"needFetch" : 0,
				"isEOF" : 0,
				"children" : [
					{
						"type" : "KEEP_MUTATIONS",
						"works" : 1,
						"yields" : 1,
						"unyields" : 1,
						"invalidates" : 0,
						"advanced" : 0,
						"needTime" : 1,
						"needFetch" : 0,
						"isEOF" : 0,
						"children" : [
							{
								"type" : "FETCH",
								"works" : 1,
								"yields" : 1,
								"unyields" : 1,
								"invalidates" : 0,
								"advanced" : 0,
								"needTime" : 1,
								"needFetch" : 0,
								"isEOF" : 0,
								"alreadyHasObj" : 0,
								"forcedFetches" : 0,
								"matchTested" : 0,
								"children" : [
									{
										"type" : "AND_HASH",
										"works" : 1,
										"yields" : 1,
										"unyields" : 1,
										"invalidates" : 0,
										"advanced" : 0,
										"needTime" : 0,
										"needFetch" : 0,
										"isEOF" : 0,
										"flaggedButPassed" : 0,
										"flaggedInProgress" : 0,
										"memUsage" : 0,
										"memLimit" : 33554432,
										"children" : [
											{
												"type" : "IXSCAN",
												"works" : 2,
												"yields" : 1,
												"unyields" : 1,
												"invalidates" : 0,
												"advanced" : 1,
												"needTime" : 1,
												"needFetch" : 0,
												"isEOF" : 0,
												"keyPattern" : "{ group: 1.0, lastModified: 1.0 }",
												"isMultiKey" : 0,
												"boundsVerbose" : "field #0['group']: [\"cn\", \"cn\"], field #1['lastModified']: [MinKey, MaxKey]",
												"yieldMovedCursor" : 0,
												"dupsTested" : 0,
												"dupsDropped" : 0,
												"seenInvalidated" : 0,
												"matchTested" : 0,
												"keysExamined" : 2,
												"children" : [ ]
											},
											{
												"type" : "IXSCAN",
												"works" : 2,
												"yields" : 1,
												"unyields" : 1,
												"invalidates" : 0,
												"advanced" : 1,
												"needTime" : 1,
												"needFetch" : 0,
												"isEOF" : 0,
												"keyPattern" : "{ clExists: 1.0, tLcLeft: 1.0 }",
												"isMultiKey" : 0,
												"boundsVerbose" : "field #0['clExists']: [true, true], field #1['tLcLeft']: [\"the national ass\", \"the national ass\"]",
												"yieldMovedCursor" : 0,
												"dupsTested" : 0,
												"dupsDropped" : 0,
												"seenInvalidated" : 0,
												"matchTested" : 0,
												"keysExamined" : 2,
												"children" : [ ]
											}
										]
									}
								]
							}
						]
					}
				]
			}
		},
		"feedback" : {
			
		},
		"filterSet" : false
	},
	{
		"details" : {
			"solution" : "(index-tagged expression tree: tree=Node\n---Leaf { clExists: 1.0, tLcLeft: 1.0 }, pos: 0\n---Leaf { lc: 1.0, group: 1.0, tLcLeft: 1.0 }, pos: 1\n---Leaf { lc: 1.0, group: 1.0, tLcLeft: 1.0 }, pos: 0\n---Leaf \n---Leaf { clExists: 1.0, tLcLeft: 1.0 }, pos: 1\n---Leaf \n---Leaf \n)"
		},
		"reason" : {
			"score" : 1.0002,
			"stats" : {
				"type" : "LIMIT",
				"works" : 1,
				"yields" : 1,
				"unyields" : 1,
				"invalidates" : 0,
				"advanced" : 0,
				"needTime" : 1,
				"needFetch" : 0,
				"isEOF" : 0,
				"children" : [
					{
						"type" : "KEEP_MUTATIONS",
						"works" : 1,
						"yields" : 1,
						"unyields" : 1,
						"invalidates" : 0,
						"advanced" : 0,
						"needTime" : 1,
						"needFetch" : 0,
						"isEOF" : 0,
						"children" : [
							{
								"type" : "FETCH",
								"works" : 1,
								"yields" : 1,
								"unyields" : 1,
								"invalidates" : 0,
								"advanced" : 0,
								"needTime" : 1,
								"needFetch" : 0,
								"isEOF" : 0,
								"alreadyHasObj" : 0,
								"forcedFetches" : 0,
								"matchTested" : 0,
								"children" : [
									{
										"type" : "AND_HASH",
										"works" : 1,
										"yields" : 1,
										"unyields" : 1,
										"invalidates" : 0,
										"advanced" : 0,
										"needTime" : 0,
										"needFetch" : 0,
										"isEOF" : 0,
										"flaggedButPassed" : 0,
										"flaggedInProgress" : 0,
										"memUsage" : 0,
										"memLimit" : 33554432,
										"children" : [
											{
												"type" : "IXSCAN",
												"works" : 2,
												"yields" : 1,
												"unyields" : 1,
												"invalidates" : 0,
												"advanced" : 1,
												"needTime" : 1,
												"needFetch" : 0,
												"isEOF" : 0,
												"keyPattern" : "{ lc: 1.0, group: 1.0, tLcLeft: 1.0 }",
												"isMultiKey" : 0,
												"boundsVerbose" : "field #0['lc']: [\"eng\", \"eng\"], field #1['group']: [\"cn\", \"cn\"], field #2['tLcLeft']: [MinKey, MaxKey]",
												"yieldMovedCursor" : 0,
												"dupsTested" : 0,
												"dupsDropped" : 0,
												"seenInvalidated" : 0,
												"matchTested" : 0,
												"keysExamined" : 2,
												"children" : [ ]
											},
											{
												"type" : "IXSCAN",
												"works" : 2,
												"yields" : 1,
												"unyields" : 1,
												"invalidates" : 0,
												"advanced" : 1,
												"needTime" : 1,
												"needFetch" : 0,
												"isEOF" : 0,
												"keyPattern" : "{ clExists: 1.0, tLcLeft: 1.0 }",
												"isMultiKey" : 0,
												"boundsVerbose" : "field #0['clExists']: [true, true], field #1['tLcLeft']: [\"the national ass\", \"the national ass\"]",
												"yieldMovedCursor" : 0,
												"dupsTested" : 0,
												"dupsDropped" : 0,
												"seenInvalidated" : 0,
												"matchTested" : 0,
												"keysExamined" : 2,
												"children" : [ ]
											}
										]
									}
								]
							}
						]
					}
				]
			}
		},
		"feedback" : {
			
		},
		"filterSet" : false
	},
	{
		"details" : {
			"solution" : "(index-tagged expression tree: tree=Node\n---Leaf \n---Leaf { lc: 1.0, group: 1.0, tLcLeft: 1.0 }, pos: 1\n---Leaf { lc: 1.0, group: 1.0, tLcLeft: 1.0 }, pos: 0\n---Leaf \n---Leaf { lc: 1.0, group: 1.0, tLcLeft: 1.0 }, pos: 2\n---Leaf { textLc: \"hashed\" }, pos: 0\n---Leaf \n)"
		},
		"reason" : {
			"score" : 1.0002,
			"stats" : {
				"type" : "LIMIT",
				"works" : 1,
				"yields" : 1,
				"unyields" : 1,
				"invalidates" : 0,
				"advanced" : 0,
				"needTime" : 1,
				"needFetch" : 0,
				"isEOF" : 0,
				"children" : [
					{
						"type" : "KEEP_MUTATIONS",
						"works" : 1,
						"yields" : 1,
						"unyields" : 1,
						"invalidates" : 0,
						"advanced" : 0,
						"needTime" : 1,
						"needFetch" : 0,
						"isEOF" : 0,
						"children" : [
							{
								"type" : "FETCH",
								"works" : 1,
								"yields" : 1,
								"unyields" : 1,
								"invalidates" : 0,
								"advanced" : 0,
								"needTime" : 1,
								"needFetch" : 0,
								"isEOF" : 0,
								"alreadyHasObj" : 0,
								"forcedFetches" : 0,
								"matchTested" : 0,
								"children" : [
									{
										"type" : "AND_SORTED",
										"works" : 1,
										"yields" : 1,
										"unyields" : 1,
										"invalidates" : 0,
										"advanced" : 0,
										"needTime" : 1,
										"needFetch" : 0,
										"isEOF" : 0,
										"flagged" : 0,
										"matchTested" : 0,
										"failedAnd_0" : 0,
										"failedAnd_1" : 0,
										"children" : [
											{
												"type" : "IXSCAN",
												"works" : 1,
												"yields" : 1,
												"unyields" : 1,
												"invalidates" : 0,
												"advanced" : 0,
												"needTime" : 1,
												"needFetch" : 0,
												"isEOF" : 0,
												"keyPattern" : "{ lc: 1.0, group: 1.0, tLcLeft: 1.0 }",
												"isMultiKey" : 0,
												"boundsVerbose" : "field #0['lc']: [\"eng\", \"eng\"], field #1['group']: [\"cn\", \"cn\"], field #2['tLcLeft']: [\"the national ass\", \"the national ass\"]",
												"yieldMovedCursor" : 0,
												"dupsTested" : 0,
												"dupsDropped" : 0,
												"seenInvalidated" : 0,
												"matchTested" : 0,
												"keysExamined" : 1,
												"children" : [ ]
											},
											{
												"type" : "IXSCAN",
												"works" : 0,
												"yields" : 1,
												"unyields" : 1,
												"invalidates" : 0,
												"advanced" : 0,
												"needTime" : 0,
												"needFetch" : 0,
												"isEOF" : 0,
												"keyPattern" : "{}",
												"isMultiKey" : 0,
												"boundsVerbose" : "field #0['textLc']: [-3202514460983226863, -3202514460983226863]",
												"yieldMovedCursor" : 0,
												"dupsTested" : 0,
												"dupsDropped" : 0,
												"seenInvalidated" : 0,
												"matchTested" : 0,
												"keysExamined" : 0,
												"children" : [ ]
											}
										]
									}
								]
							}
						]
					}
				]
			}
		},
		"feedback" : {
			
		},
		"filterSet" : false
	}
]
sprawk2:PRIMARY> db.Example.getIndexes();
[
	{
		"v" : 1,
		"key" : {
			"_id" : 1
		},
		"ns" : "jerome5.Example",
		"name" : "_id_"
	},
	{
		"v" : 1,
		"key" : {
			"terms.text" : 1
		},
		"ns" : "jerome5.Example",
		"name" : "terms.text_1"
	},
	{
		"v" : 1,
		"key" : {
			"createDate" : -1,
			"group" : 1
		},
		"ns" : "jerome5.Example",
		"name" : "createDate_-1_group_1",
		"background" : true
	},
	{
		"v" : 1,
		"key" : {
			"date" : 1
		},
		"ns" : "jerome5.Example",
		"name" : "date_1",
		"background" : true
	},
	{
		"v" : 1,
		"key" : {
			"topics" : 1
		},
		"ns" : "jerome5.Example",
		"name" : "topics_1",
		"sparse" : true,
		"background" : true
	},
	{
		"v" : 1,
		"key" : {
			"random" : 1,
			"lc" : 1
		},
		"ns" : "jerome5.Example",
		"name" : "random_1_lc_1",
		"background" : true
	},
	{
		"v" : 1,
		"key" : {
			"group" : 1,
			"lastModified" : 1
		},
		"ns" : "jerome5.Example",
		"name" : "group_1_lastModified_1",
		"background" : true
	},
	{
		"v" : 1,
		"key" : {
			"lc" : 1,
			"group" : 1,
			"tLcLeft" : 1
		},
		"ns" : "jerome5.Example",
		"name" : "lc_1_group_1_tLcLeft_1",
		"background" : true
	},
	{
		"v" : 1,
		"key" : {
			"indicesSize" : 1
		},
		"ns" : "jerome5.Example",
		"name" : "indicesSize_1",
		"background" : true
	},
	{
		"v" : 1,
		"key" : {
			"sig" : 1,
			"lc" : 1
		},
		"ns" : "jerome5.Example",
		"name" : "sig_1_lc_1",
		"background" : true
	},
	{
		"v" : 1,
		"key" : {
			"tLcLeft" : 1,
			"group" : 1,
			"trans" : 1
		},
		"ns" : "jerome5.Example",
		"name" : "tLcLeft_1_group_1_trans_1"
	},
	{
		"v" : 1,
		"key" : {
			"lc" : 1,
			"group" : 1,
			"trans" : 1,
			"wordCount" : 1,
			"pattern" : 1
		},
		"ns" : "jerome5.Example",
		"name" : "lc_1_group_1_trans_1_wordCount_1_pattern_1",
		"sparse" : true,
		"background" : true
	},
	{
		"v" : 1,
		"key" : {
			"indices.textLc" : 1,
			"group" : 1,
			"lc" : 1,
			"clExists" : 1,
			"wordCount" : 1,
			"pattern" : 1
		},
		"ns" : "jerome5.Example",
		"name" : "indices.textLc_1_group_1_lc_1_clExists_1_wordCount_1_pattern_1",
		"background" : true
	},
	{
		"v" : 1,
		"key" : {
			"u" : 1
		},
		"ns" : "jerome5.Example",
		"name" : "u_1",
		"background" : true,
		"sparse" : true
	},
	{
		"v" : 1,
		"key" : {
			"tIds" : 1,
			"lc" : 1
		},
		"ns" : "jerome5.Example",
		"name" : "tIds_1_lc_1",
		"sparse" : true,
		"background" : true
	},
	{
		"v" : 1,
		"key" : {
			"group" : 1,
			"pass" : 1
		},
		"ns" : "jerome5.Example",
		"name" : "group_1_pass_1",
		"background" : true,
		"sparse" : true
	},
	{
		"v" : 1,
		"key" : {
			"tLcHash2" : 1,
			"lc" : 1,
			"group" : 1,
			"clExists" : 1
		},
		"ns" : "jerome5.Example",
		"name" : "tLcHash2_1_lc_1_group_1_clExists_1"
	},
	{
		"v" : 1,
		"key" : {
			"length" : 1,
			"lc" : 1,
			"tLcHash2" : 1
		},
		"ns" : "jerome5.Example",
		"name" : "length_1_lc_1_tLcHash2_1"
	},
	{
		"v" : 1,
		"key" : {
			"kodoId" : 1
		},
		"ns" : "jerome5.Example",
		"name" : "kodoId_1",
		"sparse" : true,
		"background" : true
	},
	{
		"v" : 1,
		"key" : {
			"cl" : 1,
			"lc" : 1,
			"group" : 1
		},
		"ns" : "jerome5.Example",
		"name" : "cl_1_lc_1_group_1",
		"sparse" : true
	},
	{
		"v" : 1,
		"key" : {
			"v" : 1,
			"wrongLanguage" : 1,
			"group" : 1
		},
		"ns" : "jerome5.Example",
		"name" : "v_1_wrongLanguage_1_group_1"
	},
	{
		"v" : 1,
		"key" : {
			"indices.text" : 1,
			"lc" : 1,
			"group" : 1,
			"cl" : 1
		},
		"ns" : "jerome5.Example",
		"name" : "indices.text_1_lc_1_group_1_cl_1"
	},
	{
		"v" : 1,
		"key" : {
			"clExists" : 1,
			"tLcLeft" : 1
		},
		"name" : "clExists_1_tLcLeft_1",
		"ns" : "jerome5.Example",
		"sparse" : true
	},
	{
		"v" : 1,
		"key" : {
			"wl" : 1,
			"group" : 1
		},
		"name" : "wl_1_group_1",
		"ns" : "jerome5.Example",
		"sparse" : true
	},
	{
		"v" : 1,
		"key" : {
			"trans" : 1,
			"tLcLeft" : 1,
			"group" : 1,
			"lc" : 1
		},
		"name" : "trans_1_tLcLeft_1_group_1_lc_1",
		"ns" : "jerome5.Example",
		"sparse" : true
	},
	{
		"v" : 1,
		"key" : {
			"textLc" : "hashed"
		},
		"name" : "textLc_hashed",
		"ns" : "jerome5.Example"
	},
	{
		"v" : 1,
		"key" : {
			"text" : "hashed"
		},
		"name" : "text_hashed",
		"ns" : "jerome5.Example"
	},
	{
		"v" : 1,
		"key" : {
			"lb" : 1
		},
		"name" : "lb_1",
		"ns" : "jerome5.Example"
	}
]
 

Comment by Nic Cottrell (Personal) [ 20/Aug/14 ]

Here's the mongo log from this node...

Comment by Nic Cottrell (Personal) [ 20/Aug/14 ]

@Asya, here' the output you requested:

sprawk2:PRIMARY> db.Example.find( { clExists: true, lc: "eng", trans: "ara", tLcLeft: "conference", textLc: "conference", pattern: false, group: "all" }).explain(true);
{
	"cursor" : "Complex Plan",
	"n" : 0,
	"nscannedObjects" : 0,
	"nscanned" : 2,
	"nscannedObjectsAllPlans" : 0,
	"nscannedAllPlans" : 11,
	"nYields" : 1,
	"nChunkSkips" : 0,
	"millis" : 41,
	"allPlans" : [
		{
			"cursor" : "Complex Plan",
			"n" : 0,
			"nscannedObjects" : 0,
			"nscanned" : 2,
			"nChunkSkips" : 0
		},
		{
			"cursor" : "Complex Plan",
			"n" : 0,
			"nscannedObjects" : 0,
			"nscanned" : 2,
			"nChunkSkips" : 0
		},
		{
			"cursor" : "BtreeCursor lc_1_group_1_tLcLeft_1",
			"isMultiKey" : false,
			"n" : 0,
			"nscannedObjects" : 0,
			"nscanned" : 0,
			"scanAndOrder" : false,
			"indexOnly" : false,
			"nChunkSkips" : 0,
			"indexBounds" : {
				"lc" : [
					[
						"eng",
						"eng"
					]
				],
				"group" : [
					[
						"all",
						"all"
					]
				],
				"tLcLeft" : [
					[
						"conference",
						"conference"
					]
				]
			}
		},
		{
			"cursor" : "BtreeCursor tLcLeft_1_group_1_trans_1",
			"isMultiKey" : true,
			"n" : 0,
			"nscannedObjects" : 0,
			"nscanned" : 0,
			"scanAndOrder" : false,
			"indexOnly" : false,
			"nChunkSkips" : 0,
			"indexBounds" : {
				"tLcLeft" : [
					[
						"conference",
						"conference"
					]
				],
				"group" : [
					[
						"all",
						"all"
					]
				],
				"trans" : [
					[
						"ara",
						"ara"
					]
				]
			}
		},
		{
			"cursor" : "BtreeCursor lc_1_group_1_trans_1_wordCount_1_pattern_1",
			"isMultiKey" : true,
			"n" : 0,
			"nscannedObjects" : 0,
			"nscanned" : 1,
			"scanAndOrder" : false,
			"indexOnly" : false,
			"nChunkSkips" : 0,
			"indexBounds" : {
				"lc" : [
					[
						"eng",
						"eng"
					]
				],
				"group" : [
					[
						"all",
						"all"
					]
				],
				"trans" : [
					[
						"ara",
						"ara"
					]
				],
				"wordCount" : [
					[
						{
							"$minElement" : 1
						},
						{
							"$maxElement" : 1
						}
					]
				],
				"pattern" : [
					[
						false,
						false
					]
				]
			}
		},
		{
			"cursor" : "BtreeCursor group_1_pass_1",
			"isMultiKey" : false,
			"n" : 0,
			"nscannedObjects" : 0,
			"nscanned" : 1,
			"scanAndOrder" : false,
			"indexOnly" : false,
			"nChunkSkips" : 0,
			"indexBounds" : {
				"group" : [
					[
						"all",
						"all"
					]
				],
				"pass" : [
					[
						{
							"$minElement" : 1
						},
						{
							"$maxElement" : 1
						}
					]
				]
			}
		},
		{
			"cursor" : "BtreeCursor clExists_1_tLcLeft_1",
			"isMultiKey" : false,
			"n" : 0,
			"nscannedObjects" : 0,
			"nscanned" : 0,
			"scanAndOrder" : false,
			"indexOnly" : false,
			"nChunkSkips" : 0,
			"indexBounds" : {
				"clExists" : [
					[
						true,
						true
					]
				],
				"tLcLeft" : [
					[
						"conference",
						"conference"
					]
				]
			}
		},
		{
			"cursor" : "BtreeCursor trans_1_tLcLeft_1_group_1_lc_1",
			"isMultiKey" : true,
			"n" : 0,
			"nscannedObjects" : 0,
			"nscanned" : 0,
			"scanAndOrder" : false,
			"indexOnly" : false,
			"nChunkSkips" : 0,
			"indexBounds" : {
				"trans" : [
					[
						"ara",
						"ara"
					]
				],
				"tLcLeft" : [
					[
						"conference",
						"conference"
					]
				],
				"group" : [
					[
						"all",
						"all"
					]
				],
				"lc" : [
					[
						"eng",
						"eng"
					]
				]
			}
		},
		{
			"cursor" : "BtreeCursor textLc_hashed",
			"isMultiKey" : false,
			"n" : 0,
			"nscannedObjects" : 0,
			"nscanned" : 1,
			"scanAndOrder" : false,
			"indexOnly" : false,
			"nChunkSkips" : 0,
			"indexBounds" : {
				"textLc" : [
					[
						NumberLong("6337680983347079397"),
						NumberLong("6337680983347079397")
					]
				]
			}
		},
		{
			"cursor" : "Complex Plan",
			"n" : 0,
			"nscannedObjects" : 0,
			"nscanned" : 4,
			"nChunkSkips" : 0
		},
		{
			"cursor" : "Complex Plan",
			"n" : 0,
			"nscannedObjects" : 0,
			"nscanned" : 0,
			"nChunkSkips" : 0
		}
	],
	"server" : "tor:27018",
	"filterSet" : false,
	"stats" : {
		"type" : "KEEP_MUTATIONS",
		"works" : 2,
		"yields" : 1,
		"unyields" : 1,
		"invalidates" : 0,
		"advanced" : 0,
		"needTime" : 0,
		"needFetch" : 0,
		"isEOF" : 1,
		"children" : [
			{
				"type" : "FETCH",
				"works" : 1,
				"yields" : 1,
				"unyields" : 1,
				"invalidates" : 0,
				"advanced" : 0,
				"needTime" : 0,
				"needFetch" : 0,
				"isEOF" : 1,
				"alreadyHasObj" : 0,
				"forcedFetches" : 0,
				"matchTested" : 0,
				"children" : [
					{
						"type" : "AND_HASH",
						"works" : 1,
						"yields" : 1,
						"unyields" : 1,
						"invalidates" : 0,
						"advanced" : 0,
						"needTime" : 0,
						"needFetch" : 0,
						"isEOF" : 1,
						"flaggedButPassed" : 0,
						"flaggedInProgress" : 0,
						"memUsage" : 0,
						"memLimit" : 33554432,
						"children" : [
							{
								"type" : "IXSCAN",
								"works" : 2,
								"yields" : 1,
								"unyields" : 1,
								"invalidates" : 0,
								"advanced" : 1,
								"needTime" : 1,
								"needFetch" : 0,
								"isEOF" : 0,
								"keyPattern" : "{ group: 1.0, lastModified: 1.0 }",
								"isMultiKey" : 0,
								"boundsVerbose" : "field #0['group']: [\"all\", \"all\"], field #1['lastModified']: [MinKey, MaxKey]",
								"yieldMovedCursor" : 0,
								"dupsTested" : 0,
								"dupsDropped" : 0,
								"seenInvalidated" : 0,
								"matchTested" : 0,
								"keysExamined" : 2,
								"children" : [ ]
							},
							{
								"type" : "IXSCAN",
								"works" : 2,
								"yields" : 1,
								"unyields" : 1,
								"invalidates" : 0,
								"advanced" : 0,
								"needTime" : 1,
								"needFetch" : 0,
								"isEOF" : 1,
								"keyPattern" : "{ clExists: 1.0, tLcLeft: 1.0 }",
								"isMultiKey" : 0,
								"boundsVerbose" : "field #0['clExists']: [true, true], field #1['tLcLeft']: [\"conference\", \"conference\"]",
								"yieldMovedCursor" : 0,
								"dupsTested" : 0,
								"dupsDropped" : 0,
								"seenInvalidated" : 0,
								"matchTested" : 0,
								"keysExamined" : 0,
								"children" : [ ]
							}
						]
					}
				]
			}
		]
	}
}

Seems really weird to me that it would choose to start with the

{group:1, lastModified:1}

index!

Comment by J Rassi [ 19/Aug/14 ]

Hi Nic,

A value of "Complex Plan" for the "cursor" field in explain implies that the query planner chose multiple indexes for the query and performed index intersection. See the documentation for explain here: <http://docs.mongodb.org/manual/reference/method/cursor.explain/#explain.cursor>. I'd like to gather additional information to further diagnose this issue.

1. Could you upload the full mongod log for this member?

2. Could you please provide the output of running the following at the mongo shell?

var q = {clExists: true, lc: "eng", trans: "ara", tLcLeft: "conference", textLc: "conference", pattern: false, group: "all"};
db.Example.find(q).explain(true);
db.Example.find(q).itcount();
db.Example.getPlanCache().getPlansByQuery(q);
db.Example.getIndexes();

Thanks,
~ Jason Rassi

Comment by Asya Kamsky [ 19/Aug/14 ]

Could you also rerun the same query with explain(true) - that will show us more about which indexes it's considering.

Comment by Asya Kamsky [ 19/Aug/14 ]

Nic: complex query is usually index intersection. can you provide the line from the logs corresponding to one of these queries?

Comment by Nic Cottrell (Personal) [ 19/Aug/14 ]

Also, the "textLc" field has a hashed index since it sometimes exceed the 1024 byte limit.

Comment by Nic Cottrell (Personal) [ 19/Aug/14 ]

Btw, this didn't seem to be a problem in 2.6.3 - when yum-cron upgraded packages to 2.6.4 this problem started.

Generated at Thu Feb 08 03:36:29 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.