[SERVER-13002] out of memory Created: 03/Mar/14  Updated: 29/Sep/15  Resolved: 29/Sep/15

Status: Closed
Project: Core Server
Component/s: Aggregation Framework
Affects Version/s: 2.4.9
Fix Version/s: None

Type: Bug Priority: Blocker - P1
Reporter: Daniel Martinez Assignee: Amalia Hawkins
Resolution: Done Votes: 0
Labels: memory
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File trace.txt    
Operating System: ALL
Steps To Reproduce:

Collection posts: ~10 millions documents.
Total DB: ~20GB
4 CPUs
RAM: 3GB
Swap: 1GB
HDD: 100GB (25GB Used, 69G Available)

Put indexes on : author (String), permalink (String), tags (array).

Get the first five used tags:
db.posts.aggregate(
{$unwind: "$tags"},
{$group: {_id:

{id: "$_id", tag: "$tags"}

, counter:{$sum: 1}}},
{ $sort :

{ counter : -1}

},
{$limit:5}
)

Participants:

 Description   

In a virtual environment with ~20GB of data. It uses 98MB of 3GB available RAM, just one connection to DB, and using the aggregation framework.

When i execute the query bellow the Memory it is used without mesure and finally my DB it is killed.



 Comments   
Comment by Amalia Hawkins [ 04/Mar/14 ]

Glad to know it worked out. I'm closing this ticket for now.

Comment by Daniel Martinez [ 04/Mar/14 ]

Results with v2.6
Start time: ~4:45 PM
End time: ~4:55 PM
Total time: ~10 minutes

{
	"result" : [
		{
			"_id" : "sH",
			"counter" : 2898
		},
		{
			"_id" : "KI",
			"counter" : 2889
		},
		{
			"_id" : "=c",
			"counter" : 2877
		},
		{
			"_id" : "=q",
			"counter" : 2867
		},
		{
			"_id" : "al",
			"counter" : 2850
		}
	],
	"ok" : 1
}

Awesome usage of disk instead of memory (~300 MB temp files)

-----------------
Results with v2.4.9
Take ALL my swap.. and my RAM up to ~60%
My DB processs was not killed.
Start time: ~5:00 PM
End time: ~ PM

After 40 minutes I stop to watch the test.

Thanks a lot Amalia.

Comment by Amalia Hawkins [ 04/Mar/14 ]

Hey, Daniel –

In my $group operator, I've changed the _id to only correspond to a tag. That way each unique tag has its own entry, as opposed to each tag for each original _id. Did you try my query in 2.4.9? If so, what was the result?

Since 2.6 is not yet released for production use (it's only a release candidate) there is not yet complete documentation. However, try the following in 2.6:

db.posts.runCommand('aggregate', 
        {pipeline: [{$unwind: "$tags"}, 
                    {$group: {_id: "$tags", counter: {$sum: 1}}}, 
                    {$sort: {counter: -1}}, 
                    {$limit: 5}], 
        allowDiskUse: true})

Does that work?

Best,
Amalia

Comment by Daniel Martinez [ 04/Mar/14 ]

Hi Amalia, I saw not differences between my query and your query.
About v2.6 I am getting this error:

assert: command failed: {
        "errmsg" : "exception: Exceeded memory limit for $group, but didn't allow external sort. Pass allowDiskUse:true to opt in.",
        "code" : 16945,
        "ok" : 0
} : aggregate failed
Error: command failed: {
        "errmsg" : "exception: Exceeded memory limit for $group, but didn't allow external sort. Pass allowDiskUse:true to opt in.",
        "code" : 16945,
        "ok" : 0
} : aggregate failed
    at Error (<anonymous>)
    at doassert (src/mongo/shell/assert.js:11:14)
    at Function.assert.commandWorked (src/mongo/shell/assert.js:244:5)
    at DBCollection.aggregate (src/mongo/shell/collection.js:1100:12)
    at (shell):1:10
2014-03-04T15:23:58.300-0500 Error: command failed: {
        "errmsg" : "exception: Exceeded memory limit for $group, but didn't allow external sort. Pass allowDiskUse:true to opt in.",
        "code" : 16945,
        "ok" : 0
} : aggregate failed at src/mongo/shell/assert.js:13

I searched about "allowDiskUse:true", but I have not found documentation.
Then, How I can safely find the top five/ten/n used tags?

Comment by Amalia Hawkins [ 04/Mar/14 ]

Hey, Daniel,

I believe the root of this issue may be that, under version 2.4, the $group aggregation operator attempts to fit everything on the heap. This is an issue because, in your query, you're grouping on _id: {id: "$_id", tag: "$tags"}. This pulls the entire table into heap memory, creating one entry per tag per _id, which could very well be exceeding your memory and causing the crash you've encountered.

The good news is that, under 2.6, we have external sorting for $group so this should no longer be an issue. Does that make sense?

Further, I'd be happy to help you reformat your query to avoid this issue in the 2.4.9 version. Are you trying to find the top five most used tags across all posts? In that case, you could instead format your query like so:

db.posts.aggregate({$unwind: "$tags"},
                   {$group: {_id: "$tags", counter: {$sum: 1}}},
                   {$sort: {counter: -1}},
                   {$limit:5})

Best,
Amalia

Generated at Thu Feb 08 03:30:20 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.