[SERVER-44964] [4.2] Calculate index build memory usage per key based on BSONObj buffer capacity, not size Created: 05/Dec/19  Updated: 29/Oct/23  Resolved: 25/Dec/19

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.6.0, 4.0.0, 4.2.0
Fix Version/s: 4.2.3

Type: Bug Priority: Major - P3
Reporter: Louis Williams Assignee: Eric Milkie
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Problem/Incident
Related
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Execution Team 2019-12-30
Participants:
Linked BF Score: 0

 Description   

Index builds limit their memory usage with the parameter maxIndexBuildMemoryUsageMegabytes, which defaults to 500MB. The memory usage calculation depends on the BSONObj accurately reporting its memory usuage in memUsageForSorter().

BSONObj reports its memory usage as the size of the object, but not the capacity of the buffer holding that object. This is problematic because index builds can use significantly more memory than they they should.

We should change the imlementation of memUsageForSorter() to report the size of the underlying SharedBuffer, in a very similar way to  what we did for in SERVER-42931.

 

 



 Comments   
Comment by Githook User [ 23/Dec/19 ]

Author:

{'name': 'Eric Milkie', 'email': 'milkie@mongodb.com', 'username': 'milkie'}

Message: SERVER-44964 calculate index build memory usage per key based on BSONObj buffer capacity instead of object size
Branch: v4.2
https://github.com/mongodb/mongo/commit/38e05a18e716da5020d8f01f37e8bd8cf8f769b1

Generated at Thu Feb 08 05:07:29 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.