Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-23980

$graphLookup should spill to disk if allowDiskUse is specified

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major - P3
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 3.5 Desired
    • Component/s: Aggregation Framework
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Sprint:
      Query 2017-01-23

      Description

      $graphLookup should spill to disk when the size of the frontier and the visited set exceeds the maximum memory usage. In particular, DocumentSourceGraphLookUp::checkMemoryUsage() should be modified to spill if the size of the two data structures is too large. Second, any place where the frontier or visited set is iterated will need to be updated to use a file iterator if it has spilled to disk.

        Issue Links

          Activity

          Hide
          brandon.newell Brandon Newell added a comment -

          Confirmed issue continues to exist in 3.4.0-rc3

          Even when adding $limit to pipeline:

          db.friends.aggregate( [
             { $limit : NumberInt(50) },
             {
                $graphLookup: {
                   from: "friends",
                   startWith: "$friends.first_name",
                   connectFromField: "friends.first_name",
                   connectToField: "first_name",
                   as: "connectedFriends"
                }
             }
          ], {
               allowDiskUse: true
                }).pretty()
          

          Show
          brandon.newell Brandon Newell added a comment - Confirmed issue continues to exist in 3.4.0-rc3 Even when adding $limit to pipeline: db.friends.aggregate( [ { $limit : NumberInt(50) }, { $graphLookup: { from: "friends", startWith: "$friends.first_name", connectFromField: "friends.first_name", connectToField: "first_name", as: "connectedFriends" } } ], { allowDiskUse: true }).pretty()
          Hide
          benjamin.murphy Benjamin Murphy (Inactive) added a comment - - edited

          I don't believe this has been implemented yet, hence, allowDiskUsage does not affect $graphLookup. Also, $limit won't affect the depth of the recursion of the $graphLookup; you can only restrict that using maxDepth.

          Show
          benjamin.murphy Benjamin Murphy (Inactive) added a comment - - edited I don't believe this has been implemented yet, hence, allowDiskUsage does not affect $graphLookup. Also, $limit won't affect the depth of the recursion of the $graphLookup; you can only restrict that using maxDepth.
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'cswanson310', u'name': u'Charlie Swanson', u'email': u'charlie.swanson@mongodb.com'}

          Message: SERVER-23980 Flush out DocumentComparator library.
          Branch: master
          https://github.com/mongodb/mongo/commit/e50c923ff402a4b2276e62e053eb5326c06b8976

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'cswanson310', u'name': u'Charlie Swanson', u'email': u'charlie.swanson@mongodb.com'} Message: SERVER-23980 Flush out DocumentComparator library. Branch: master https://github.com/mongodb/mongo/commit/e50c923ff402a4b2276e62e053eb5326c06b8976

            People

            • Votes:
              5 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:

                  Agile