Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-6273

aggregation tracks memory usage for a single $sort stage, not the entire pipeline

    • Type: Icon: Improvement Improvement
    • Resolution: Won't Fix
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Aggregation Framework
    • Labels:
      None

      Observed behavior: A memory assertion is not triggered if the combined memory usage of several stages exceeds 10% of physical ram, so long as no single sort stage exceeds 10% of physical ram.
      Expected behavior: A memory assertion is triggered if the combined memory usage exceeds 10% of physical ram.

      Right now memory usage is only tracked separately for each sort stage.

      Test

      c = db.c;
      c.drop();
      
      big = new Array( 10000 + 1 ).toString();
      
      for( i = 0; i < 40000; ++i ) {
          c.save( { x:i, y:i, z:i, b:big } );
      }
      
      printjson( c.aggregate( { $sort:{ x:1 } }, { $sort:{ y:1 } }, { $sort:{ z:1 } }, { $limit:1 } ) );
      

      The test ran on a machine where the total data size was between 5% and 10% of physical ram. Each sort stage separately warned of memory usage exceeding 5% of physical ram, but the combined memory usage of the sort stages was not calculated or reported on.

      Mon Jul  2 06:55:58 [conn6] warning: warning, 5% of physical RAM used for [ { $sort: { x: 1 } } ]
      warning, 5% of physical RAM used for [ { $sort: { y: 1 } } ]
      warning, 5% of physical RAM used for [ { $sort: { z: 1 } } ]
      

            Assignee:
            mathias@mongodb.com Mathias Stearn
            Reporter:
            aaron Aaron Staple
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: