[SERVER-44095] S2 key generation vector should use a smaller type Created: 18/Oct/19  Updated: 29/Oct/23  Resolved: 10/Mar/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 4.4.0-rc0, 4.7.0

Type: Bug Priority: Major - P3
Reporter: Louis Williams Assignee: Louis Williams
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.4
Sprint: Execution Team 2020-03-09, Execution Team 2020-03-23
Participants:
Linked BF Score: 49

 Description   

S2, "2dsphere", indexes generate keys usingĀ a vector of KeyString::HeapBuilder. Unfortunately, this type is very large (608 bytes), as compared to the previously used BSONObj (16 bytes). This type is large because of the StackBufBuilder used for the TypeBits member, which when not allocated on the stack, adds an additional 512 bytes to the size of this type. Most of this space is unused.

This causes problems for 2dsphere indexes, which when building a compound key where the first field is an array, need to allocate as many KeyString::HeapBuilders as there are elements in the array. If the array is very large, say 1500000 elements, then the minimum memory requirement to generate the keys is close to 900MB.

We should find a way to use a smaller type in this vector, KeyString::Value (24 bytes) for example, and resort to copying it as necessary.

An alternative would be to make TypeBits support the BufBuilder instead of the StackBufBuilder, but that would only reduce the HeapBuilder type to around 96 bytes.



 Comments   
Comment by Githook User [ 11/Mar/20 ]

Author:

{'name': 'Louis Williams', 'username': 'louiswilliams', 'email': 'louis.williams@mongodb.com'}

Message: SERVER-44095 Initialize TypeBits buffer to a smaller size

This lowers the memory overhead of index key generation by reducing the
initial buffer size in TypeBits from 512 to 8 bytes.

(cherry picked from commit 4994a4809269c30a28cdbb686c8c876fad7b163a)
Branch: v4.4
https://github.com/mongodb/mongo/commit/2027183c69d0dac503722251ef204e02346c721e

Comment by Githook User [ 09/Mar/20 ]

Author:

{'name': 'Louis Williams', 'username': 'louiswilliams', 'email': 'louis.williams@mongodb.com'}

Message: SERVER-44095 Initialize TypeBits buffer to a smaller size

This lowers the memory overhead of index key generation by reducing the
initial buffer size in TypeBits from 512 to 8 bytes.
Branch: master
https://github.com/mongodb/mongo/commit/4994a4809269c30a28cdbb686c8c876fad7b163a

Comment by Connie Chen [ 17/Jan/20 ]

Check on geo-indexes, request from Atlas

Generated at Thu Feb 08 05:05:00 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.