[SERVER-43816] Push $text and $sort with $meta in the sort pattern into the PlanStage layer Created: 03/Oct/19  Updated: 29/Oct/23  Resolved: 18/Dec/19

Status: Closed
Project: Core Server
Component/s: Aggregation Framework, Querying
Affects Version/s: None
Fix Version/s: 4.3.3

Type: Improvement Priority: Major - P3
Reporter: David Storch Assignee: David Storch
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Documented
is documented by DOCS-13305 Investigate changes in SERVER-43816: ... Closed
Duplicate
is duplicated by SERVER-40253 Pipelines with text search queries sh... Closed
Related
is related to SERVER-7568 Aggregation framework favors non-bloc... Closed
Backwards Compatibility: Minor Change
Sprint: Query 2019-12-16, Query 2019-12-30
Participants:

 Description   

When a $sort stage is at the beginning of the pipeline, or close to the beginning, the aggregation subsystem attempts to execute it using the underlying PlanStage query execution tree. We are trying to push increasingly more execution into the PlanStage layer in order to move towards having a single, unified execution engine which will be easier to extend and maintain.

There is a special case where we don't push down $sort if it has "textScore" or "randVal" $meta expressions: see https://github.com/mongodb/mongo/blob/8ccaacce937ae51deef27c41e977aca0511fb338/src/mongo/db/pipeline/pipeline_d.cpp#L102-L120.

For "textScore", this restriction stems from the fact that the find and agg systems have different validity rules for $meta sort by text score. The agg subsystem permits $meta:textScore with no matching $meta projection:

> db.c.drop()
> db.c.createIndex({f: "text"})
{
	"createdCollectionAutomatically" : true,
	"numIndexesBefore" : 1,
	"numIndexesAfter" : 2,
	"commitQuorum" : 1,
	"ok" : 1
}
> db.c.aggregate([{$match: {$text: {$search: "foo"}}}, {$sort: {f: {$meta: "textScore"}}}])
// Success!

The find subsystem, on the other hand, requires a matching "textScore" $meta projection:

>  db.c.find({$text: {$search: "foo"}}, {}).sort({f: {$meta: "textScore"}})
Error: error: {
	"ok" : 0,
	"errmsg" : "must have $meta projection for all $meta sort keys",
	"code" : 2,
	"codeName" : "BadValue"
}
> db.c.find({$text: {$search: "foo"}}, {f: {$meta: "textScore"}}).sort({f: {$meta: "textScore"}})
// Success!

We should eliminate this restriction, which would allow us to always push down "textScore" $meta sort.

The situation for "randVal" needs some investigation, but note SERVER-43666. The ability to sort by "randVal" has existed internally for some time to support $sample, though I don't think we intend to support it as available to applications.



 Comments   
Comment by Githook User [ 18/Dec/19 ]

Author:

{'name': 'David Storch', 'email': 'david.storch@mongodb.com', 'username': 'dstorch'}

Message: SERVER-43816 Fix typo in error message.
Branch: master
https://github.com/mongodb/mongo/commit/58bae64c6c66c577328e4171dc96889bcc2157a5

Comment by David Storch [ 18/Dec/19 ]

Included in the changes for this ticket are several behavior changes related to the $meta and $natural operators.

First, this change tightens the validation for $natural sort and $natural hint. Previous versions of the server would permit $natural in a compound key pattern or compound sort pattern. However, it is not meaningful to combine $natural with another non-$natural field. After this change, this will result in an error with an appropriate error message:

MongoDB Enterprise > db.c.find().hint({a: 1, $natural: 1})
Error: error: {
	"ok" : 0,
	"errmsg" : "Cannot include '$natural' in compoound hint: { a: 1.0, $natural: 1.0 }",
	"code" : 2,
	"codeName" : "BadValue"
}
MongoDB Enterprise > db.c.find().hint({$natural: 1, b: 1})
Error: error: {
	"ok" : 0,
	"errmsg" : "Cannot include '$natural' in compoound hint: { $natural: 1.0, b: 1.0 }",
	"code" : 2,
	"codeName" : "BadValue"
}
MongoDB Enterprise > db.c.find().sort({a: 1, $natural: 1})
Error: error: {
	"ok" : 0,
	"errmsg" : "Cannot include '$natural' in compoound sort: { a: 1.0, $natural: 1.0 }",
	"code" : 2,
	"codeName" : "BadValue"
}
MongoDB Enterprise > db.c.find().sort({$natural: 1, b: 1})
Error: error: {
	"ok" : 0,
	"errmsg" : "Cannot include '$natural' in compoound sort: { $natural: 1.0, b: 1.0 }",
	"code" : 2,
	"codeName" : "BadValue"
}

Second, this change removes the restriction that $natural sort cannot be used in a find command against a view. Previous versions would require such an operation to be specified as a $natural hint when querying a view. As of this change, both $natural sort and $natural hint will work as expected against a view:

MongoDB Enterprise > db.createCollection("c")
{ "ok" : 1 }
MongoDB Enterprise > db.createView("v", "c", [])
{ "ok" : 1 }
 
// Both of these queries against view "v" run successfully.
MongoDB Enterprise > db.v.find().hint({$natural: 1})
MongoDB Enterprise > db.v.find().sort({$natural: 1})

Third, this change tightens the logic for producing errors when a $meta:"textScore" expression is present, but the predicate does not include $text. In previous versions, a find command specifying a "textScore" $meta-sort and "textScore" $meta projection would not raise an error if the $text search predicate was absent. Now, the server correctly returns an error with an appropriate error message:

MongoDB Enterprise > db.c.find({}, {a: {$meta: "textScore"}}).sort({a: {$meta: "textScore"}})
Error: error: {
	"ok" : 0,
	"errmsg" : "query requires text score metadata, but it is not available",
	"code" : 40218,
	"codeName" : "Location40218"
}

Finally, there are changes which relax the validation around "textScore" $meta projection and "textScore" $meta-sort for the find command. Previous versions of the server required a "textScore" $meta projection on the same field as a "textScore" $meta-sort whenever the $meta-sort is specified. The changes under this ticket relax this requirement. Find commands can now specify $meta-sort and $meta-projection with different field names, can specify $meta-sort without $meta projection, can specify $meta projection without $meta-sort, and can sort on the same field that is $meta-projected:

MongoDB Enterprise > db.c.createIndex({a: "text"})
{
	"numIndexesBefore" : 1,
	"numIndexesAfter" : 2,
	"commitQuorum" : 1,
	"createdCollectionAutomatically" : true,
	"ok" : 1
}
MongoDB Enterprise > db.c.insert({a: "one"})
WriteResult({ "nInserted" : 1 })
MongoDB Enterprise > db.c.insert({a: "one two"})
WriteResult({ "nInserted" : 1 })
 
// $meta projection uses field name "score", but $meta-sort uses "other". This is now legal.
MongoDB Enterprise > db.c.find({$text: {$search: "one two"}}, {score: {$meta: "textScore"}}).sort({other: {$meta: "textScore"}})
{ "_id" : ObjectId("5df99ff448333b145081b9bf"), "a" : "one two", "score" : 1.5 }
{ "_id" : ObjectId("5df99ff148333b145081b9be"), "a" : "one", "score" : 1.1 }
 
// $meta-sort without $meta projection. This is now legal.
MongoDB Enterprise > db.c.find({$text: {$search: "one two"}}).sort({score: {$meta: "textScore"}})
{ "_id" : ObjectId("5df99ff448333b145081b9bf"), "a" : "one two" }
{ "_id" : ObjectId("5df99ff148333b145081b9be"), "a" : "one" }
 
// $meta projection without $meta-sort. This is now legal.
MongoDB Enterprise > db.c.find({$text: {$search: "one two"}}, {score: {$meta: "textScore"}})
{ "_id" : ObjectId("5df99ff148333b145081b9be"), "a" : "one", "score" : 1.1 }
{ "_id" : ObjectId("5df99ff448333b145081b9bf"), "a" : "one two", "score" : 1.5 }
 
// Regular (non-$meta) sort on the non-existent "score" field, and then assign the result of a "textScore" $meta projection to the
// "score" field. This is now legal.
MongoDB Enterprise > db.c.find({$text: {$search: "one two"}}, {score: {$meta: "textScore"}}).sort({other: 1})
{ "_id" : ObjectId("5df99ff148333b145081b9be"), "a" : "one", "score" : 1.1 }
{ "_id" : ObjectId("5df99ff448333b145081b9bf"), "a" : "one two", "score" : 1.5 }

Comment by Githook User [ 18/Dec/19 ]

Author:

{'name': 'David Storch', 'email': 'david.storch@mongodb.com', 'username': 'dstorch'}

Message: SERVER-43816 Push $text and $meta:"textScore" sort into the PlanStage layer.

This change involves unifying the behavior of find and
agg for validation of $meta:"textScore". In particular, find
operations no longer require a "textScore" $meta projection
in order to specify a "textScore" $meta sort. This brings
find into alignment with agg, which never had such a
restriction. It is also now legal for a find command to
sort on the field overridden by a $meta:"textScore" projection
without specifying the $meta operator in the sort pattern.

In addition:

Generated at Thu Feb 08 05:04:11 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.