[SERVER-43993] Improve Document memory usage during blocking sort Created: 14/Oct/19  Updated: 02/Apr/20  Resolved: 02/Apr/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Justin Seyster Assignee: Ian Boros
Resolution: Duplicate Votes: 0
Labels: qexec-team
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-45763 Optimize extractKeyFast() fallback path Closed
Operating System: ALL
Sprint: Query 2019-11-04, Query 2019-11-18, Query 2019-12-02, Query 2019-12-16, Query 2019-12-30, Query 2020-01-13, Query 2020-01-27, Query 2020-02-10, Query 2020-03-23, Query 2020-04-06
Participants:

 Description   

The changes in SERVER-42565 lead sort key generation to perform field lookups on Document objects instead of raw BSON. These lookups fault field values into each Document's internal field cache. In the worst case, a single lookup can fault in every field, causing every Document in the blocking sort to use up twice as much memory.

This worst case occurred in BF-14987, where a blocking sort that was supposed to fit in memory instead exceeds the memory limit for sorts.

We have two possible approaches in mind to fix this:
1) Always perform projection before a blocking sort (but after sort key generation). After projection is done, there is no performance benefit from keeping the original BSON around. We can pull all of a Document's BSON into its cache and destroy the underlying BSON.

2) Add a non-faulting lookup function to Document and use that for sort key generation.



 Comments   
Comment by Ian Boros [ 02/Apr/20 ]

Given that we've implemented suggestion (2) under SERVER-45763, I'm closing this as a duplicate.

Comment by David Storch [ 20/Feb/20 ]

Justin is moving onto other priorities, so I'm marking this to be re-triaged.

Comment by David Storch [ 18/Oct/19 ]

justin.seyster can you investigate possible approaches for this problem during this sprint?

Generated at Thu Feb 08 05:04:42 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.