-
Type: Bug
-
Resolution: Unresolved
-
Priority: Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
Labels:
-
Storage Execution
-
Execution Team 2024-05-13
Right now, we allow double maxValidateMemoryUsageMB during the second phase of validation. It appears like we use double the memory limit but we actually double count the number of index keys in the first phase. We can update the comment to make this more clear. If a user sets maxValidateMemoryUsage to their max available memory like in atlas, validation can OOM because our limit calculation undercounts actual memory usage/ignores other metadata for each key. We should respect maxValidateMemoryUsageMB as the memory limit and consider doubling the server default.
We want to consider accounting for other metadata that is stored along with each key during validation when calculating the estimated memory usage.