Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-64021

Lookup stage poor performance

    • ALL
    • Hide

      Create a collection with 100K documents.

      Create a collection with any number of documents.

      Join the second collection on the first collection.

      Show
      Create a collection with 100K documents. Create a collection with any number of documents. Join the second collection on the first collection.

      We switch from a SQL database to MongoDb for our latest cloud application and the performance benefits we received were spectacular. We also followed the MongoDb guidelines and used document embedding when possible. Currently none of our standard CRUD operations require any form of join and are all getting executed incredibly fast (our data sets range from 100K to 2M documents).

       

      Howerver, in order for us to generate certain reports we need to perform certain joins between collections. The MongoDb C# driver that we are using translates the LINQ join query to a lookup stage. The lookup stage performs a search to the joined collection for every entry of the initial collection. We knew that the "join" performance was lacking on MongoDb, and should be avoided, but requiring 20 seconds to join a collection with 200K documents with a collection with 100K documents was totally unexpected.

       

      This is a known issue and is acknowledged by the MongoDb team 7 years ago and 2 solutions were proposed.

      [SERVER-21312|SERVER-21312 $lookup should batch query requests - MongoDB Jira]

      [SERVER-21284|SERVER-21284 $lookup should cache query results - MongoDB Jira]

       

      We would like to know if you are planning to resolve this issue or if this issue is not possible to be resolved (in order for us to try to resolve it using external methods). Granted that this issue has such a devastating performance impact and has not been resolved for almost 7 years we are lead to believe that it's the later case. 

       

      Thanks in advance!

            Assignee:
            edwin.zhou@mongodb.com Edwin Zhou
            Reporter:
            paplabros@gmail.com Labros Papadopoulos
            Votes:
            5 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: