[SERVER-43096] Pre-allocate smaller buffer for "distinct" commands Created: 30/Aug/19  Updated: 29/Oct/23  Resolved: 08/Oct/19

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 4.2.1, 4.3.1

Type: Improvement Priority: Major - P3
Reporter: Bruce Lucas (Inactive) Assignee: James Wahlin
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Backwards Compatibility: Fully Compatible
Backport Requested:
v4.2, v4.0
Sprint: Query 2019-10-07, Query 2019-10-21
Participants:
Case:

 Description   

We allocate a 16 MB buffer to hold the results of every "distinct" command, no matter how big or small the result is. This can create a performance problem: every buffer is allocated directly from and returned directly to the tcmalloc pageheap, which is moderately expensive, and is a serial bottleneck.

Also, tcmalloc decommits memory (returns it to the o/s) at a rate more-or-less proportional to the rate at which memory is returned to the pageheap, so this high rate of memory returned to the pageheap can create a high rate of decommitting and the re-commiting memory, which is a very expensive operation.

We could consider pre-allocating a much smaller buffer here and allowing it to grow as needed.



 Comments   
Comment by Githook User [ 05/Nov/19 ]

Author:

{'name': 'James Wahlin', 'username': 'jameswahlin', 'email': 'james.wahlin@mongodb.com'}

Message: SERVER-43096 Pre-allocate smaller buffer for "distinct" commands

(cherry picked from commit 22256673a9db8f9cf44bdb149fe20a599ecb7e9a)
(cherry picked from commit 1e29423eedf119d055b60aad28c086a0cf5460ac)
Branch: v4.0
https://github.com/mongodb/mongo/commit/b37a2dd6d573e126a07cf97c7c3bc295cfb679e5

Comment by Githook User [ 15/Oct/19 ]

Author:

{'name': 'James Wahlin', 'username': 'jameswahlin', 'email': 'james.wahlin@mongodb.com'}

Message: SERVER-43096 Pre-allocate smaller buffer for "distinct" commands

(cherry picked from commit 22256673a9db8f9cf44bdb149fe20a599ecb7e9a)
Branch: v4.2
https://github.com/mongodb/mongo/commit/1e29423eedf119d055b60aad28c086a0cf5460ac

Comment by Githook User [ 08/Oct/19 ]

Author:

{'username': 'jameswahlin', 'email': 'james.wahlin@mongodb.com', 'name': 'James Wahlin'}

Message: SERVER-43096 Pre-allocate smaller buffer for "distinct" commands
Branch: master
https://github.com/mongodb/mongo/commit/22256673a9db8f9cf44bdb149fe20a599ecb7e9a

Generated at Thu Feb 08 05:02:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.