[SERVER-23980] $graphLookup should spill to disk if allowDiskUse is specified Created: 28/Apr/16 Updated: 27/Dec/23 |
|
| Status: | Backlog |
| Project: | Core Server |
| Component/s: | Aggregation Framework |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Benjamin Murphy | Assignee: | Backlog - Query Execution |
| Resolution: | Unresolved | Votes: | 24 |
| Labels: | asya, query-44-grooming | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||
| Assigned Teams: |
Query Execution
|
||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||
| Sprint: | Query 2017-01-23 | ||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||||||||||||||
| Description |
|
$graphLookup should spill to disk when the size of the frontier and the visited set exceeds the maximum memory usage. In particular, DocumentSourceGraphLookUp::checkMemoryUsage() should be modified to spill if the size of the two data structures is too large. Second, any place where the frontier or visited set is iterated will need to be updated to use a file iterator if it has spilled to disk. |
| Comments |
| Comment by Rai Deepak [ 19/Sep/22 ] | ||||||||||||||
|
When this feature expected to rollout? | ||||||||||||||
| Comment by Charlie Swanson [ 11/Dec/18 ] | ||||||||||||||
|
This ticket has a fixVersion of "Backlog" which indicates we still plan to do it, but have no particular timeline in which we plan to do so. Thank you for voicing your interest. If you haven't done so already, I'd encourage you to vote for and watch the ticket for future updates. For further context we did start to work on this a little while ago (looks like some time around Dec 2016), but it became clear it was not straightforward and there were interesting design decisions to be made. It soon got dropped in favor of other priorities. | ||||||||||||||
| Comment by Krzysztof Kukiełka [ 11/Dec/18 ] | ||||||||||||||
|
I can see a TODO -https://github.com/mongodb/mongo/blob/master/src/mongo/db/pipeline/document_source_graph_lookup.cpp#L393 - in your code, but when will it be ready? As mulamba said: any ETA or ups? | ||||||||||||||
| Comment by Mukul Lamba [ 16/Oct/18 ] | ||||||||||||||
|
It is an important feature to add true graph functionalities to mongo, do we have ETA or any updates on this? Thanks | ||||||||||||||
| Comment by Githook User [ 16/Dec/16 ] | ||||||||||||||
|
Author: {u'username': u'cswanson310', u'name': u'Charlie Swanson', u'email': u'charlie.swanson@mongodb.com'}Message: SERVER-23980 Flush out DocumentComparator library. | ||||||||||||||
| Comment by Benjamin Murphy [ 15/Nov/16 ] | ||||||||||||||
|
I don't believe this has been implemented yet, hence, allowDiskUsage does not affect $graphLookup. Also, $limit won't affect the depth of the recursion of the $graphLookup; you can only restrict that using maxDepth. | ||||||||||||||
| Comment by Brandon Newell [ 15/Nov/16 ] | ||||||||||||||
|
Confirmed issue continues to exist in 3.4.0-rc3 Even when adding $limit to pipeline:
|