Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-7037

Error in M/R -- 'value too large to reduce'

    • Type: Icon: Bug Bug
    • Resolution: Incomplete
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 2.0.0, 2.2.0
    • Component/s: MapReduce
    • Labels:
      None
    • ALL

      When running a map/reduce job on a collection containing a large number of entity for a single key in the map/reduce operation, the computation fails because of the size of the object fed into the reduce.
      In our case the reduce accumulates the values encountered in an array. The finalize would after trim down these values and apply the final operations to the final object. This accumulation process cannot be filtered before and needs all the informations.

      Code:
      =====

      m = function(){
          /**
           * @description Mapper
           * @return 
           **/
          if( this._attributes ){
      	emit(
      	    this._attributes.entityid, 
      	    {
      	    'entity': this,
      	    'deltas':[]
      	    }
      	    );
          } else if( this.RelatedEntityId )
          {
      	emit( this.RelatedEntityId, 
      	      {
      		  'entity': undefined,
      		  'deltas': [this]
      	      }
      	    );
          }
      };
      
      
      r = function(key, values){
          /**
           * @description Reducer function
           * @param key
           * @param values Values associated with the given key
           * @return map containing up to 3 fields: deltas with a list of deltas, entity with the entity, key which contains the key
           **/
      
          var reducedPackage = { 'deltas' : [], 
      			   'entity' : undefined };
          for( indexValue in values ){
      	currentValue = values[indexValue];
      	if( currentValue.entity ){
      	    reducedPackage.entity = currentValue.entity;
      	}
      	reducedPackage.deltas = reducedPackage.deltas.concat( currentValue.deltas );
          }
          return reducedPackage;
      };
      

      Additional information:
      =======================
      Avg Object Size: 82 Kb, Size of the database: 60 Mb, Storage: 80 Mb
      Object Count: 801
      Number of MapReduce keys: 1

            Assignee:
            tad Tad Marshall
            Reporter:
            mete_dizioglu@ultimatesoftware.com Mete Dizioglu
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: