[SERVER-14466] Support projecting the $id or $ref for a DBRef() field Created: 04/Jul/14  Updated: 27/Mar/20  Resolved: 22/Nov/19

Status: Closed
Project: Core Server
Component/s: Aggregation Framework
Affects Version/s: 2.6.0
Fix Version/s: 4.3.3

Type: Bug Priority: Major - P3
Reporter: Stennie Steneker (Inactive) Assignee: Ian Boros
Resolution: Done Votes: 20
Labels: QFB, asya, eng-m, expression, query-44-grooming, read-only-views, usability
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Duplicate
is duplicated by SERVER-22005 Aggregation pipeline does not allow r... Closed
is duplicated by SERVER-34992 FieldPath field names may not start w... Closed
is duplicated by SERVER-43129 Test $elemMatch projection of DBRef f... Closed
Problem/Incident
Related
related to SERVER-30365 How to exclude (sub)fields starting w... Backlog
related to SERVER-8436 Aggregation pipeline cannot process s... Backlog
is related to SERVER-30417 add expression to get value by keynam... Closed
Backwards Compatibility: Minor Change
Sprint: Query 2019-11-18, Query 2019-12-02
Participants:
Linked BF Score: 39

 Description   

DBRef fields are only supported as passthrough values. It would be useful to be able to extract the $id and $ref values for manipulation in a pipeline.

*edit* it's also possible to use full DBRef values for comparisons.



 Comments   
Comment by Roberto Como [ 27/Mar/20 ]

Are there any plans to merge this fix in a production (4.2.X ) build ?

Comment by Githook User [ 21/Nov/19 ]

Author:

{'email': 'ian.boros@mongodb.com', 'name': 'Ian Boros', 'username': 'puppyofkosh'}

Message: SERVER-14466 test use of DBRef fields in aggregation and find() expressions
Branch: master
https://github.com/mongodb/mongo/commit/92d0ebf34f39f82f430869273ae751c1d97ec960

Comment by Githook User [ 21/Nov/19 ]

Author:

{'name': 'Mathew Robinson', 'username': 'chasinglogic', 'email': 'mathew.robinson@mongodb.com'}

Message: Revert "SERVER-14466 test use of DBRef fields in aggregation and find() expressions"
Branch: master
https://github.com/mongodb/mongo/commit/81318c61ba8b2f88f1e43359fc4b5ffbc7d1934b

Comment by Githook User [ 20/Nov/19 ]

Author:

{'name': 'Ian Boros', 'username': 'puppyofkosh', 'email': 'ian.boros@mongodb.com'}

Message: SERVER-14466 test use of DBRef fields in aggregation and find() expressions
Branch: master
https://github.com/mongodb/mongo/commit/0c398983e0d63ac13b03e8136e0126af537ca311

Comment by Asya Kamsky [ 17/Nov/19 ]

nburuzul

You cannot use a field added in a stage till the following stage - in other words new field is added to documents as a result of your $addFields, therefore you cannot refer to it in the same stage that's defining it.

These types of questions are better resolved in one of the support forums.

Comment by naresh buruzula [ 13/Nov/18 ]

I am trying to use new filed (created using $addFields) inside path.

$addFields: {

   "identifier": {$ifNull: ["$requests.identifiers.id", ""]},
   "key" : {$ifNull: ["$outputs.*$identifier*.response.value", ""] }

}

I want to use the identifier field from fist step in the path of the second step (while creating the key field).

but I don't get $identifier evaluated and get the below err.

err: FieldPath field names may not start with '$'.

Is this allowed in mongodb 3.6?

Comment by Asya Kamsky [ 13/Feb/18 ]

Note that while projection of individual sub-components of DBRef doesn't currently work in aggregation (other than via workaround mentioned above) it is possible to use full DBRef in comparison operations as I show here

Comment by Josef Sábl [ 13/Dec/17 ]

I've hit this wall due to this bug.

Comment by Asya Kamsky [ 15/Jun/17 ]

For a more generalized conversion of an object "foo" into an object that has all the leading "$" stripped from the key names, this expression should do it:

db.dbref.aggregate({$addFields:{"newfoo":
     {$arrayToObject:{$map:{
          input:{$objectToArray:"$foo"}, 
          in:{
             k:{$cond:[ 
                     {$eq:[{"$substrCP":["$$this.k",0,1]},{$literal:"$"}]},
                     {$substrCP:["$$this.k",1,{$strLenCP:"$$this.k"}]},
                     "$$this.k"
             ]},
             v:"$$this.v"
           }
         }}}
}})

This takes field "$foo" that's an object and makes "newfoo" the same object but with key names "$name" stipped of first character. So

"foo" : DBRef("bar", "_id")

becomes

"newfoo" : { "ref" : "bar", "id" : "_id" }

Comment by Andreas B. [ 14/Jun/17 ]

Thanks charlie.swanson! That looks hacky, but I guess it'll have to do until the issue is resolved.

Comment by Charlie Swanson [ 14/Jun/17 ]

Hi a.braun,

If you're just looking to get the values out, this should work:

> db.foo.insert({_id: 0, ref: DBRef("bar", 0)})
WriteResult({ "nInserted" : 1 })
> db.foo.find()
{ "_id" : 0, "ref" : DBRef("bar", 0) }
> db.foo.aggregate([{$project: {x: {$objectToArray: "$$ROOT.ref"}}}])
{ "_id" : 0, "x" : [ { "k" : "$ref", "v" : "bar" }, { "k" : "$id", "v" : 0 } ] }
> db.foo.aggregate([{$project: {ref: {$let: {vars: {refParts: {$objectToArray: "$$ROOT.ref"}}, in: "$$refParts.v"}}}}])
{ "_id" : 0, "ref" : [ "bar", 0 ] }

It uses the new $objectToArray expression, introduced in SERVER-18794 (version 3.4.4)

Comment by Andreas B. [ 14/Jun/17 ]

showed me how to use $objectToArray + $filter + $map + $arrayElemAt

For the uninitiated, could you please post the workaround? I'm quite interested in knowing how to do this.

Comment by Siyuan Zhou [ 13/Jun/17 ]

david.storch, thanks for asking! charlie.swanson showed me how to use $objectToArray + $filter + $map + $arrayElemAt to work around this issue, so we are not blocked by this issue.

Comment by Asya Kamsky [ 08/Jun/17 ]

When change notification is first stage in aggregation any data manipulation that's done in that pipeline may need to project or check/use the value of "o.$__" field ($set, $unset, etc).

Comment by David Storch [ 08/Jun/17 ]

siyuan.zhou, is this required for the $changeNotification work, or do you have a workaround? I'd like to know if you need work from the query team so that we plan our time accordingly.

Comment by Siyuan Zhou [ 07/Jun/17 ]

Aggregation over local.oplog.rs also will benefit from this when referring o.$set to get the changed fields of updates.

Comment by Asya Kamsky [ 19/Apr/17 ]

Same for distinct. db.coll.distinct("foo.$ref")

Comment by Asya Kamsky [ 19/Apr/17 ]

Also can cause an error when doing projection on DBRef in a view.

db.dbref.find({},{"foo.$id":1})
{ "_id" : ObjectId("58f794e568275eb7a0d08db5"), "foo" : { "$id" : "_id" } }
db.dbrefview.find({},{"foo.$id":1})
Error: error: {
	"ok" : 0,
	"errmsg" : "FieldPath field names may not start with '$'.",
	"code" : 16410,
	"codeName" : "Location16410"
}

Comment by Asya Kamsky [ 21/Jul/16 ]

Also affects $graphLookup the same way.

Comment by Igor Dianov [X] [ 14/Feb/16 ]

This also affects the $lookup stage introduced in 3.2 which was designed to do joins to other documents which are quite frequently referenced by DBRef objects. For example:

db.getCollection('Positions').aggregate([{$lookup: {from: "Employees", localField: "employeeRef.$id", foreignField: "_id", as: "employee" }}])

assert: command failed:

{ "ok" : 0, "errmsg" : "FieldPath field names may not start with '$'.", "code" : 16410 }

: aggregate failed
_getErrorWithCode@src/mongo/shell/utils.js:23:13
doassert@src/mongo/shell/assert.js:13:14
assert.commandWorked@src/mongo/shell/assert.js:266:5
DBCollection.prototype.aggregate@src/mongo/shell/collection.js:1215:5
@(shell):1:1

Comment by Asya Kamsky [ 04/Feb/16 ]

From description of duplicate ticket:

While the $match stage does allow references to DBRef fields (i.e. $id, $ref, $db), users cannot refer to those fields in $project and $group stages (I haven't tested others). I don't see a reason why this would conflict with field path and system variables, since the "$" in question is not a prefix. Could the assertions be relaxed to allow these DBRef fields to be used in pipelines?

I'll note that while storing a referenced document's ID in a basic field (or array for reference-many relationships) may be the general advice for users, we've never formally deprecated use of DBRef (not to be confused with DBPointer, which was deprecated in BSON). The DBRef convention is still widely used in ODM libraries.

Generated at Thu Feb 08 03:34:57 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.