Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-48390

group with $accumulator complains memory exceeds 100MBs on smaller than 96MBs collection

    XMLWordPrintable

    Details

    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Backport Requested:
      v4.4
    • Sprint:
      Query 2020-08-24

      Description

      I'm not sure how $accumulator contributes to tracking group size but it appears to be way overcounting when passed documents (unless it's allocating a huge amount to JS in general?)

      db.scores.aggregate([{$match:{game:/^G[123]/}},{$count:"c"}])
      { "c" : 368061 }
      db.scores.aggregate([{$match:{game:/^G[123]/}},{$group:{_id:0, size:{$sum:{$bsonSize:"$$ROOT"}}}}])
      { "_id" : 0, "size" : 30215379 }  /* 28GBs */
      db.scores.aggregate([{$match:{game:/^G[123]/}}, {$group:{_id:"$game", top2: { $accumulator: { init: function() {  return null;   },  accumulateArgs: [ [1,2,3,4,5,"$player","$score"] ],                     accumulate: function(state, val) {    return state;     },                     merge: function(state1, state2) {                         return state1;  },      finalize: function(state) {   return state;   }   } }  } },{$count:"c"}])
      { "c" : 33 }
      /* as soon as I pass a document as args */
      db.scores.aggregate([{$match:{game:/^G[123]/}}, {$group:{_id:"$game", top2: { $accumulator: { init: function() {  return null;   },  accumulateArgs: [ {score:"$score"} ],                     accumulate: function(state, val) {    return state;     },                     merge: function(state1, state2) {                         return state1;  },      finalize: function(state) {   return state;   }   } }  } },{$count:"c"}])
      Error: command failed: {
      	"ok" : 0,
      	"errmsg" : "Exceeded memory limit for $group, but didn't allow external sort. Pass allowDiskUse:true to opt in.",
      	"code" : 292,
      	"codeName" : "QueryExceededMemoryLimitNoDiskUseAllowed"
      } : aggregate failed :
      _getErrorWithCode@src/mongo/shell/utils.js:25:13
      doassert@src/mongo/shell/assert.js:18:14
      _assertCommandWorked@src/mongo/shell/assert.js:618:17
      assert.commandWorked@src/mongo/shell/assert.js:708:16
      DB.prototype._runAggregate@src/mongo/shell/db.js:266:5
      DBCollection.prototype.aggregate@src/mongo/shell/collection.js:1012:12
      @(shell):1:1
      /* with limit */
      db.scores.aggregate([{$match:{game:/^G[123]/}},{$limit:335000}, {$group:{_id:"$game", top2: { $accumulator: { init: function() {  return null;   },  accumulateArgs: [ {score:"$score"} ],                     accumulate: function(state, val) {    return state;     },                     merge: function(state1, state2) {                         return state1;  },      finalize: function(state) {   return state;   }   } }  } },{$count:"c"}])
      { "c" : 30 }
      db.scores.aggregate([{$match:{game:/^G[123]/}},{$limit:340000}, {$group:{_id:"$game", top2: { $accumulator: { init: function() {  return null;   },  accumulateArgs: [ {score:"$score"} ],                     accumulate: function(state, val) {    return state;     },                     merge: function(state1, state2) {                         return state1;  },      finalize: function(state) {   return state;   }   } }  } },{$count:"c"}])
      Error: command failed: {
      	"ok" : 0,
      	"errmsg" : "Exceeded memory limit for $group, but didn't allow external sort. Pass allowDiskUse:true to opt in.", ...
      

      Collection stats:

      db.scores.aggregate({$collStats:{storageStats:{scale:1024*1024}}},{$project:{"storageStats.wiredTiger":0,"storageStats.indexDetails":0}}).pretty()
      {
      	"ns" : "agg.scores",
      	"host" : "asyas-mbp-4.lan:27017",
      	"localTime" : ISODate("2020-05-23T18:44:17.107Z"),
      	"storageStats" : {
      		"size" : 95,
      		"count" : 1130151,
      		"avgObjSize" : 88,
      		"storageSize" : 31,
      		"freeStorageSize" : 0,
      		"capped" : false,
      		"nindexes" : 2,
      		"indexBuilds" : [ ],
      		"totalIndexSize" : 51,
      		"totalSize" : 82,
      		"indexSizes" : {
      			"_id_" : 19,
      			"game_1_score_-1" : 32
      		},
      		"scaleFactor" : 1048576
      	}
      }
      

        Attachments

          Activity

            People

            Assignee:
            arun.banala Arun Banala
            Reporter:
            asya Asya Kamsky
            Participants:
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: