[SERVER-58411] Deep find nested objects in return documents. Created: 10/Jul/21  Updated: 06/May/22  Resolved: 29/Mar/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Major - P3
Reporter: Gerrie van Wyk Assignee: Chris Harris
Resolution: Won't Do Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-22497 Allow from field of $lookup and $grap... Backlog
Sprint: QE 2021-08-09, QE 2021-08-23, QE 2021-09-06, QE 2021-09-20, QE 2021-10-04, QE 2021-10-18, QE 2021-11-01, QE 2021-11-15, QE 2021-11-29, QE 2021-12-13, QE 2021-12-27, QE 2022-01-10, QE 2022-01-24
Participants:

 Description   

Not sure how feasible this is, but working with relational data in MongoDB is always a pain since finding nested documents requires writing an aggregation using $lookup, $unwind, etc. 

Just thought of a convenient implementation to solve this. If only ObjectIds included a collection id, then you could call a "deep find" method which fetches all nested objects where it encounters an ObjectId in the return documents. 

Other considerations here would be infinite loop protection and a max depth feature.



 Comments   
Comment by Chris Harris [ 29/Mar/22 ]

Hi gvw@lowdefy.com,

Thanks again for filing this request.

As outlined by Asya above - your desired functionality is mostly achievable today by using DBRefs.  Alternatively you could consider using Views to store the aggregation pipeline in the database itself, which can simplify how the client application(s) expresses read requests.  Based on this, we do not currently have any plans to implement this functionality and will therefore proceed to close this ticket.

It is also worth noting that, as with any database system, appropriately modeling your data is important for long-term success.  We have a variety of resources available on the topic, but please do reach out to the community on the forums if you have questions or would like more information in this regard.

Best,
Chris

Comment by Asya Kamsky [ 20/Jul/21 ]

gvw@lowdefy.com I'm not sure I understand the original problem. You said:

> finding nested documents requires writing an aggregation using $lookup, $unwind

Generally finding matching nested documents does not require $unwind neither does returning matching subdocuments.

$lookup is not really related to finding documents but rather joining together two collections, but that also shouldn't require extra stages (obviously it requires aggregation since it's a stage in agg).

With the example you provide it looks like you are using the term "nested documents" to mean "related documents in a different collection(s)" rather than what we usually mean (which is having an array of documents inside the document)... And the mechanism you are describing looks similar to DBRef which is meant to be used by the driver(s) or other external frameworks to resolve such relationships (and has some advantages but also disadvantages).

Note that we cannot resolve such references in the server because we check collection access permissions at parse time and here collection is specified inside the data dynamically. SERVER-22497 asks for ability to specify it as an expression (including a dynamic field in the document) but our AUTH model is the reason we cannot implement it that way.

Comment by Gerrie Van wyk [ 16/Jul/21 ]

Hi @Eric

Thanks for looking at this. Perhaps let me try and explain what I feel would be an "nice to have" with an example:

Painpoint

Working with relational data in mongo is challenging simply because in order to fetch the document and nested documents, one needs to run an aggregation and use $lookup, handle $unwind where required. 

Solution

As a dev, it would be great to have:

const res_A = collection_A.insertOne({ a: 1});
const res_B = collection_B.insertOne({ b: 2, a: res_A.insertedId });
 
const res = collection_B.deepFind({ _id: res_B._id });
// where res is:
// res = { _id: ObjectID('abc'), b: 2, a: { _id: ObjectID('xyz'), a: 1} }

From a user perspective this would make it really easy to work with relational data, instead of writing aggregations to deal with all this.

Currently, I suspect the above would not be possible because ObjectId does not contain any information for which collection it is created / stored in so the find method has no idea where to look to the relational document. Hence you need to provide the collection and lined field in the $lookup stage for the aggregation.

As an future idea, if ObjectIDs contained a short collection id, then the mongo server would know where to go and find the any nested relational document - thus a deepFind method can be added to resolve the nested documents linked using the improved ObjectIDs. 

Obviously I have no idea how complex such an implementation would be if even possible, however, from an end user / dev perspective it would make working with relational data in MongoDB sooo nice.

 

Comment by Eric Sedor [ 15/Jul/21 ]

Thanks gvw@lowdefy.com. I read this as a request for some form of dynamic, recursive $lookup with the pain point being that it's a challenge to keep complex document structures and complex aggregation pipelines in agreement. I'll pass this on to an appropriate team to consider. But please feel free to clarify or elaborate on your use-case.

Eric

Generated at Thu Feb 08 05:44:27 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.