[SERVER-6273] aggregation tracks memory usage for a single $sort stage, not the entire pipeline Created: 02/Jul/12  Updated: 10/Dec/14  Resolved: 17/Sep/13

Status: Closed
Project: Core Server
Component/s: Aggregation Framework
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Aaron Staple Assignee: Mathias Stearn
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-6862 improve usability of aggregation sort... Closed
Related
related to SERVER-6157 Refactor BSONObjExtSort Closed
Participants:

 Description   

Observed behavior: A memory assertion is not triggered if the combined memory usage of several stages exceeds 10% of physical ram, so long as no single sort stage exceeds 10% of physical ram.
Expected behavior: A memory assertion is triggered if the combined memory usage exceeds 10% of physical ram.

Right now memory usage is only tracked separately for each sort stage.

Test

c = db.c;
c.drop();
 
big = new Array( 10000 + 1 ).toString();
 
for( i = 0; i < 40000; ++i ) {
    c.save( { x:i, y:i, z:i, b:big } );
}
 
printjson( c.aggregate( { $sort:{ x:1 } }, { $sort:{ y:1 } }, { $sort:{ z:1 } }, { $limit:1 } ) );

The test ran on a machine where the total data size was between 5% and 10% of physical ram. Each sort stage separately warned of memory usage exceeding 5% of physical ram, but the combined memory usage of the sort stages was not calculated or reported on.

Mon Jul  2 06:55:58 [conn6] warning: warning, 5% of physical RAM used for [ { $sort: { x: 1 } } ]
warning, 5% of physical RAM used for [ { $sort: { y: 1 } } ]
warning, 5% of physical RAM used for [ { $sort: { z: 1 } } ]



 Comments   
Comment by Daniel Pasette (Inactive) [ 17/Sep/13 ]

This is not true anymore because agg should release memory as soon as a new sort stage is entered. Any single stage should not use more than 100mb, and no more than two stages will hold data at one time, regardless of the number of sorts.

Comment by Aaron Staple [ 02/Jul/12 ]

Assigning to release candidate for triage.

Generated at Thu Feb 08 03:11:13 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.