Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-4566

In new aggregation framework, $sort, $limit in the pipeline seems loading all the matched data into memory. When we tried to improve the performance by leveraging multi thread aggregation, this makes it much slower than single thread

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major - P3
    • Resolution: Fixed
    • 2.1.1
    • None
    • Aggregation Framework
    • None
    • OSX 10.6, Java, mongodb is built from Dec. 15 master branch.
    • OS X

    Description

      Overall new aggregation framework is faster than map-reduce in our application. But when I tried to further improve the performance by sending multiple aggregation commands in multiple threads, then get the final aggregation done in app space, I noticed each thread will load lot of data to memory which makes each command take minutes to finish. (I do have index on sort key)

      Attachments

        Issue Links

          depends on

          Improvement - An improvement or enhancement to an existing feature or task. SERVER-3832 aggregation: early $sort should be optimized to use an index if possible

          • Major - P3 - Major loss of function.
          • Closed
          is depended on by

          Improvement - An improvement or enhancement to an existing feature or task. SERVER-4507 aggregation: optimize $group to take advantage of sorted sequences

          • Major - P3 - Major loss of function.
          • Backlog

          Activity

            People

              cwestin Chris Westin
              xiaofeng Xiaofeng Wu
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: