[SERVER-83615] Treat maxValidateMemoryUsageMB as an absolute limit Created: 27/Nov/23  Updated: 29/Jan/24

Status: Open
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Shin Yee Tan Assignee: Backlog - Storage Execution Team
Resolution: Unresolved Votes: 0
Labels: storex-ranked
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
Assigned Teams:
Storage Execution
Participants:

 Description   

Right now, we allow double maxValidateMemoryUsageMB during the second phase of validation. It appears like we use double the memory limit but we actually double count the number of index keys in the first phase. We can update the comment to make this more clear. If a user sets maxValidateMemoryUsage to their max available memory like in atlas, validation can OOM because our limit calculation undercounts actual memory usage/ignores other metadata for each key. We should respect maxValidateMemoryUsageMB as the memory limit and consider doubling the server default.

We want to consider accounting for other metadata that is stored along with each key during validation when calculating the estimated memory usage.


Generated at Thu Feb 08 06:52:41 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.