Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-43993

Improve Document memory usage during blocking sort

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • ALL
    • Query 2019-11-04, Query 2019-11-18, Query 2019-12-02, Query 2019-12-16, Query 2019-12-30, Query 2020-01-13, Query 2020-01-27, Query 2020-02-10, Query 2020-03-23, Query 2020-04-06

      The changes in SERVER-42565 lead sort key generation to perform field lookups on Document objects instead of raw BSON. These lookups fault field values into each Document's internal field cache. In the worst case, a single lookup can fault in every field, causing every Document in the blocking sort to use up twice as much memory.

      This worst case occurred in BF-14987, where a blocking sort that was supposed to fit in memory instead exceeds the memory limit for sorts.

      We have two possible approaches in mind to fix this:
      1) Always perform projection before a blocking sort (but after sort key generation). After projection is done, there is no performance benefit from keeping the original BSON around. We can pull all of a Document's BSON into its cache and destroy the underlying BSON.

      2) Add a non-faulting lookup function to Document and use that for sort key generation.

            Assignee:
            ian.boros@mongodb.com Ian Boros
            Reporter:
            justin.seyster@mongodb.com Justin Seyster
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: