[SERVER-23980] $graphLookup should spill to disk if allowDiskUse is specified Created: 28/Apr/16  Updated: 27/Dec/23

Status: Backlog
Project: Core Server
Component/s: Aggregation Framework
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Benjamin Murphy Assignee: Backlog - Query Execution
Resolution: Unresolved Votes: 24
Labels: asya, query-44-grooming
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-23725 Implement $graphLookup. Closed
Duplicate
is duplicated by SERVER-38632 allowDiskUse(true) does not work as e... Closed
Related
related to SERVER-84427 Add a query knob to configure $graphL... Backlog
is related to SERVER-74370 100MB stage memory limit for $graphLo... Backlog
Assigned Teams:
Query Execution
Backwards Compatibility: Fully Compatible
Sprint: Query 2017-01-23
Participants:
Case:

 Description   

$graphLookup should spill to disk when the size of the frontier and the visited set exceeds the maximum memory usage. In particular, DocumentSourceGraphLookUp::checkMemoryUsage() should be modified to spill if the size of the two data structures is too large. Second, any place where the frontier or visited set is iterated will need to be updated to use a file iterator if it has spilled to disk.



 Comments   
Comment by Rai Deepak [ 19/Sep/22 ]

When this feature expected to rollout?

Comment by Charlie Swanson [ 11/Dec/18 ]

This ticket has a fixVersion of "Backlog" which indicates we still plan to do it, but have no particular timeline in which we plan to do so. Thank you for voicing your interest. If you haven't done so already, I'd encourage you to vote for and watch the ticket for future updates. 

For further context we did start to work on this a little while ago (looks like some time around Dec 2016), but it became clear it was not straightforward and there were interesting design decisions to be made. It soon got dropped in favor of other priorities.

Comment by Krzysztof Kukiełka [ 11/Dec/18 ]

I can see a TODO -https://github.com/mongodb/mongo/blob/master/src/mongo/db/pipeline/document_source_graph_lookup.cpp#L393 - in your code, but when will it be ready? As mulamba said: any ETA or ups?

Comment by Mukul Lamba [ 16/Oct/18 ]

It is an important feature to add true graph functionalities to mongo, do we have ETA or any updates on this? Thanks

Comment by Githook User [ 16/Dec/16 ]

Author:

{u'username': u'cswanson310', u'name': u'Charlie Swanson', u'email': u'charlie.swanson@mongodb.com'}

Message: SERVER-23980 Flush out DocumentComparator library.
Branch: master
https://github.com/mongodb/mongo/commit/e50c923ff402a4b2276e62e053eb5326c06b8976

Comment by Benjamin Murphy [ 15/Nov/16 ]

I don't believe this has been implemented yet, hence, allowDiskUsage does not affect $graphLookup. Also, $limit won't affect the depth of the recursion of the $graphLookup; you can only restrict that using maxDepth.

Comment by Brandon Newell [ 15/Nov/16 ]

Confirmed issue continues to exist in 3.4.0-rc3

Even when adding $limit to pipeline:

db.friends.aggregate( [
   { $limit : NumberInt(50) },
   {
      $graphLookup: {
         from: "friends",
         startWith: "$friends.first_name",
         connectFromField: "friends.first_name",
         connectToField: "first_name",
         as: "connectedFriends"
      }
   }
], {
     allowDiskUse: true
      }).pretty()

Generated at Thu Feb 08 04:05:01 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.