Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-39166

$graphLookup should force a pipeline to split in sharded cluster

    XMLWordPrintable

    Details

    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Backport Requested:
      v4.0, v3.6, v3.4
    • Sprint:
      Query 2019-02-11

      Description

      The $graphLookup stage does not inherit from NeedsMergerDocumentSource, which means that it is eligible to execute in parallel on all shards. This doesn't work because it will only ever do the searching locally. Instead, it should force itself to split the pipeline and execute in the merging half of the pipeline so that it can always run on the primary shard for the database.

      The $graphLookup StageConstraints are correct in saying that it must run on the primary shard, but this would only apply if some other stage before $graphLookup forced the pipeline to split.

      For example, see the following running against a mongos:

      mongos> db.adminCommand({enableSharding: "test"})
      ...
      mongos> db.adminCommand({shardCollection: "test.foo", key: {_id: "hashed"}})
      ...
      mongos> db.foo.insert([{}, {}, {}, {}])
      ...
      mongos> db.bar.insert({_id: 1, x: 1})
      WriteResult({ "nInserted" : 1 })
      mongos> db.foo.aggregate([{$project: {l_x: {$literal: 1}}}, {$graphLookup: {from: "bar", startWith: "$l_x", connectFromField: "x", connectToField: "_id", as: "res"}}])
      { "_id" : ObjectId("5c48f159a3c842122a8dec28"), "l_x" : 1, "res" : [ ] }  // Uh oh, these should have the same results as below.
      { "_id" : ObjectId("5c48f159a3c842122a8dec29"), "l_x" : 1, "res" : [ ] }
      { "_id" : ObjectId("5c48f159a3c842122a8dec26"), "l_x" : 1, "res" : [ { "_id" : 1, "x" : 1 } ] }
      { "_id" : ObjectId("5c48f159a3c842122a8dec27"), "l_x" : 1, "res" : [ { "_id" : 1, "x" : 1 } ] }
      

        Attachments

          Activity

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: