Concurrent (async) eager loading

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Unknown
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Ticket MONGOID-5445 introduced "load_async" introduced asynchronous query execution.

      It would be nice if eager loading of async queries could leverage the async mechanism as well. This would massively turbocharge complex queries which iterate over large collections, which is a typical use case for reporting.

      For example, consider the following:

      Pet.includes(owner: [:house, :account], :friends).load_async

      This could use multiple threads in a cascading (producer-consumer) fashion, e.g.

      • Thread A loads Pets
      • For each GETMORE batch of 1000 pets produced by Thread A:
        • Thread B loads owners
        • Thread C loads friends
      • For each GETMORE batch owners produced by Thread B:
        • Thread D loads houses
        • Thread E loads accounts
      • Thread A does not finally yield the batch of pets until associated threads B/C/D/E have all returned results.

       

              Assignee:
              Unassigned
              Reporter:
              Johnny Shields
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: