parallel $lookup large output (many matches for single batch) scenario

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Atlas Stream Processing Engine
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      1. better handle the scenario where the subop (like LookUp) produces tons of output for a single input batch. This can happen in $lookup if there are tons of matches. It will complicate things quite a bit so I'd rather deal with it later.
      2. track memory in this scenario
      3. Also, in the testWithManyUnwindMatches test in lookup.js, there is a test which has a $lookup followed by a $unwind with 10k matches. Make this $100k matches. This was running very slow for some reason.
         

            Assignee:
            Jada Lilleboe
            Reporter:
            Jada Lilleboe
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: