[SERVER-13002] out of memory Created: 03/Mar/14 Updated: 29/Sep/15 Resolved: 29/Sep/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Aggregation Framework |
| Affects Version/s: | 2.4.9 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Blocker - P1 |
| Reporter: | Daniel Martinez | Assignee: | Amalia Hawkins |
| Resolution: | Done | Votes: | 0 |
| Labels: | memory | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Operating System: | ALL |
| Steps To Reproduce: | Collection posts: ~10 millions documents. Put indexes on : author (String), permalink (String), tags (array). Get the first five used tags: , counter:{$sum: 1}}}, }, |
| Participants: |
| Description |
|
In a virtual environment with ~20GB of data. It uses 98MB of 3GB available RAM, just one connection to DB, and using the aggregation framework. When i execute the query bellow the Memory it is used without mesure and finally my DB it is killed. |
| Comments |
| Comment by Amalia Hawkins [ 04/Mar/14 ] | |||||||||||||||||||||||||
|
Glad to know it worked out. | |||||||||||||||||||||||||
| Comment by Daniel Martinez [ 04/Mar/14 ] | |||||||||||||||||||||||||
|
Results with v2.6
Awesome usage of disk instead of memory (~300 MB temp files) ----------------- After 40 minutes I stop to watch the test. Thanks a lot Amalia. | |||||||||||||||||||||||||
| Comment by Amalia Hawkins [ 04/Mar/14 ] | |||||||||||||||||||||||||
|
Hey, Daniel – In my $group operator, I've changed the _id to only correspond to a tag. That way each unique tag has its own entry, as opposed to each tag for each original _id. Did you try my query in 2.4.9? If so, what was the result? Since 2.6 is not yet released for production use (it's only a release candidate) there is not yet complete documentation. However, try the following in 2.6:
Does that work? Best, | |||||||||||||||||||||||||
| Comment by Daniel Martinez [ 04/Mar/14 ] | |||||||||||||||||||||||||
|
Hi Amalia, I saw not differences between my query and your query.
I searched about "allowDiskUse:true", but I have not found documentation. | |||||||||||||||||||||||||
| Comment by Amalia Hawkins [ 04/Mar/14 ] | |||||||||||||||||||||||||
|
Hey, Daniel, I believe the root of this issue may be that, under version 2.4, the $group aggregation operator attempts to fit everything on the heap. This is an issue because, in your query, you're grouping on _id: {id: "$_id", tag: "$tags"}. This pulls the entire table into heap memory, creating one entry per tag per _id, which could very well be exceeding your memory and causing the crash you've encountered. The good news is that, under 2.6, we have external sorting for $group so this should no longer be an issue. Does that make sense? Further, I'd be happy to help you reformat your query to avoid this issue in the 2.4.9 version. Are you trying to find the top five most used tags across all posts? In that case, you could instead format your query like so:
Best, |