[SERVER-60137] Improve Buffer Allocation for Common BSON-related Sizes Created: 22/Sep/21  Updated: 29/Oct/23  Resolved: 05/Oct/21

Status: Closed
Project: Core Server
Component/s: Query Execution
Affects Version/s: None
Fix Version/s: 5.1.0-rc0

Type: Improvement Priority: Major - P3
Reporter: Mohammad Dashti (Inactive) Assignee: Mohammad Dashti (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-36346 Large memory consumption per pending ... Closed
Related
is related to SERVER-60115 Fix the Buffer Builder to Correctly u... Closed
is related to SERVER-60116 Fix Excessive Pointer Arithmetics in ... Closed
Backwards Compatibility: Fully Compatible
Sprint: QE 2021-10-04, QE 2021-10-18
Participants:

 Description   

Currently, when we are requesting a piece of memory that might contain a BSON object, it usually requires BSONObjMaxUserSize bytes (i.e., 16MB) plus a residual amount of memory for its wrapper data structure, which is BSONObjMaxInternalSize bytes (i.e., 16MB+16KB). Then, we ask the buffer builder to allocate this much memory, it tries to allocate an amount of memory that's a power of two. In this case, it allocates 32MB of memory, which is usually a waste of resources.

In this ticket, based on the discussions with bernard.gorman, david.storch and redbeard0531, we want to change the buffer builder logic in the grow_reallocate to allocate BSONObjMaxInternalSize bytes if the requested re-allocated amount is anything between BSONObjMaxUserSize and BSONObjMaxInternalSize bytes.



 Comments   
Comment by Vivian Ge (Inactive) [ 06/Oct/21 ]

Updating the fixversion since branching activities occurred yesterday. This ticket will be in rc0 when it’s been triggered. For more active release information, please keep an eye on #server-release. Thank you!

Comment by Mohammad Dashti (Inactive) [ 05/Oct/21 ]

redbeard0531 Yeah, it'll be done in SERVER-60115

Comment by Mathias Stearn [ 05/Oct/21 ]

This change didn't adjust for the 8 bytes used for the SharedBuffer::Holder. Is that coming in a later commit?

Comment by Githook User [ 05/Oct/21 ]

Author:

{'name': 'Mohammad Dashti', 'email': 'mdashti@gmail.com', 'username': 'mdashti'}

Message: SERVER-60137 Improved the buffer allocation for the common BSON-related sizes
Branch: master
https://github.com/mongodb/mongo/commit/de1400af129cbe6db8abe12fa39f1e8a96c24c40

Comment by Mathias Stearn [ 22/Sep/21 ]

BSONObjMaxInternalSize isn't enough. That is only the maximum size of a single bson object, however this allocation size is for the message, which has an additional envelope structure. I think 16M+64K aught to be enough to not worry about going over, while still having a very low relative overhead (well under 1%).

Generated at Thu Feb 08 05:49:03 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.