[SERVER-57116] Investigate improvements of JavaScript OOM issue with $function expression Created: 21/May/21  Updated: 06/Dec/22

Status: Backlog
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Milena Ivanova Assignee: Backlog - Query Execution
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
Assigned Teams:
Query Execution
Participants:

 Description   

A ticket investigated an OOM error of JavaScript engine when computing $function expression. The conclusion was that the cause of the problem is the garbage collection of the JavaScript engine in combination with the heap limit of 100MB.

There were two workarounds proposed:

1) process data in large collections in batches;

2) increase the JS heap size. This is done on-prem via

db.adminCommand({setParameter: 1, "internalQueryJavaScriptHeapSizeLimitMB" : 1000});

This workaround is not applicable for users on Atlas due to authorisation issues. It is also not described in the public documentation.

 

The purpose of this task is to investigate further improvements of the JS OOM issue, so that users don't experience it or have easier work-around. The following ideas were proposed and need further investigation:

1) increase the default size of JS heap;

2) add a heapSize parameter to $function expression which may allow for increasing the JS heap size per expression base (idea of kevin.pulo)

3) make the GC more active, for instance by decreasing the current threshold that triggers GC from 80% to 70% of memory usage.

4) provide better information for the issue:

  • include the set parameter in the public documentation: advised agains by PM, since it creates opportunity for experimentation with sensitive parameters which are better protected.
  • improve the error message for JS OOM, for instance add a hint to use the 'set parameter' command or the heapSize parameter if 2) is implemented.

 

These ideas could be helpful also for the coming project to upgrade the JavaScript engine.

 


Generated at Thu Feb 08 05:41:01 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.