[SERVER-62086] Update HashAggStage's default maximum memory usage to the bigger value comparable to the DocumentSourceGroup's Created: 16/Dec/21  Updated: 29/Oct/23  Resolved: 16/Dec/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 5.3.0, 5.2.0-rc2

Type: Bug Priority: Major - P3
Reporter: Yoon Soo Kim Assignee: Yoon Soo Kim
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v5.2
Sprint: QE 2021-12-27
Participants:
Linked BF Score: 150

 Description   

Here are investigation results for BF-23718:

  1. They are the same issue which is related to the new SBE spilling behavior
  2. The pipeline in both issues has $group stage on $obj.str group-by key in which all documents has different values and number of documents is 200. The HashAgg checks whether the estimated memory usage is over the allowed maximum memory usage per every 100 groups. So, at the 200th group, the HashAgg realizes that the memory usage over the configured value and errors out when allowDiskUse is off.
  3. Then why doesn’t mongod-5.0 fail? Because we take the classic engine DocumentSourceGroup’s code path at mongod-5.0 and the default maximum memory usage is 100 * 1024 * 1024 (==100MB). But in SBE HashAggStage, the default maximum memory usage is 1024 * 1024.

DocumentSourceGroup’s default maximum memory usage:
https://github.com/mongodb/mongo/blob/master/src/mongo/db/query/query_knobs.idl#L330-L338
HashAggStage’s default maximum memory usage:
https://github.com/mongodb/mongo/blob/master/src/mongo/db/query/query_knobs.idl#L527-L536

So, my recommendation is to update HashAggStage’s default maximum memory usage to 100 * 1024 * 1024.



 Comments   
Comment by Githook User [ 17/Dec/21 ]

Author:

{'name': 'Yoonsoo Kim', 'email': 'yoonsoo.kim@mongodb.com', 'username': 'yun-soo'}

Message: SERVER-62086 Increase `HashAggStage`'s maximum allowed memory usage before spill
Branch: v5.2
https://github.com/mongodb/mongo/commit/51cba4e2277ba895127f13bf7098fc29fc72c2cd

Comment by Githook User [ 16/Dec/21 ]

Author:

{'name': 'Yoonsoo Kim', 'email': 'yoonsoo.kim@mongodb.com', 'username': 'yun-soo'}

Message: SERVER-62086 Increase `HashAggStage`'s maximum allowed memory usage before spill
Branch: master
https://github.com/mongodb/mongo/commit/298f70be324278def91cd5350a358cf3405146bc

Generated at Thu Feb 08 05:54:08 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.