[SERVER-8287] Large array result causes v8 shell to run out of memory Created: 23/Jan/13 Updated: 11/Jul/16 Resolved: 04/Feb/13 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | JavaScript, Shell |
| Affects Version/s: | 2.3.2 |
| Fix Version/s: | 2.4.0-rc1 |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Ben Becker | Assignee: | Ben Becker |
| Resolution: | Done | Votes: | 0 |
| Labels: | javascript, shell | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||
| Issue Links: |
|
||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||
| Operating System: | ALL | ||||||||
| Participants: | |||||||||
| Description |
|
The following script causes v8 to run out of memory with the current resource constraints. When the constraints are removed, the shell uses ~750mb of resident memory. Subsequent runs of the GC only free ~640mb.
|
| Comments |
| Comment by auto [ 04/Feb/13 ] |
|
Author: {u'date': u'2013-02-04T01:14:22Z', u'name': u'Ben Becker', u'email': u'ben.becker@10gen.com'}Message: |
| Comment by Ben Becker [ 23/Jan/13 ] |
|
Patch fixes the issue, even with our current resource constraints. Mem usage in the shell (vs. NodeJS) is nearly identical – 138mb. It may be worth converting these to native functions at some point to further reduce memory consumption. |
| Comment by Aaron Heckmann [ 23/Jan/13 ] |
|
Attached is the diff used to produce better results - res mem sat around 130MB consistently. Summary: string concatenation in v8 is very bad. Use an array and .join() it instead. |
| Comment by Ben Becker [ 23/Jan/13 ] |
|
As aheckmann pointed out, string concatenation (+=) appears to be very expensive in terms of memory and CPU time. Commenting out the loop that replaces escaped characters in tojson() decreased total mem consumption from 750mb to 200mb. |
| Comment by Ben Becker [ 23/Jan/13 ] |
|
The source of memory consumption is our tojson() and tojsonObject() JS functions (called from shellPrintHelper(_lastres_) in dbshell.cpp). Retrieving the results without displaying them only results in ~60mb resident memory. Resident mem ramps up to ~750mb while executing the tojson/tojsonObject functions, and the GC starts thrashing; moving objects from young to old space while compacting. In debug mode, printing the results takes 32 seconds (vs. ~1.5 seconds to retrieve results). Replacing the shellPrintHelper with a much simpler function completes in less than a second with no noticeable increase in memory consumption. This function doesn't do all the required work though. Cursory scanning of the tojson() functions reviled several expensive O(n) operations which are likely to produce thousands of 'young' objects for each result in this test. For example, every string is iterated over to replace escaped characters. Another example is counting an Object's properties before displaying them. |
| Comment by Ben Becker [ 23/Jan/13 ] |
|
One work-around is to remove (or significantly increase) the resource limits we set for each isolate. In a local test, the mongo client uses ~750mb of resident memory when resource constraints are lifted. gc() reduces consumption to 112mb (much higher than the 10.1mb we start with). Seems like we're allocating too much memory, and GC isn't freeing everything it can, even after multiple runs. |