Error in M/R -- 'value too large to reduce'

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Incomplete
    • Priority: Major - P3
    • None
    • Affects Version/s: 2.0.0, 2.2.0
    • Component/s: MapReduce
    • None
    • ALL
    • None
    • 3
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      When running a map/reduce job on a collection containing a large number of entity for a single key in the map/reduce operation, the computation fails because of the size of the object fed into the reduce.
      In our case the reduce accumulates the values encountered in an array. The finalize would after trim down these values and apply the final operations to the final object. This accumulation process cannot be filtered before and needs all the informations.

      Code:
      =====

      m = function(){
          /**
           * @description Mapper
           * @return 
           **/
          if( this._attributes ){
      	emit(
      	    this._attributes.entityid, 
      	    {
      	    'entity': this,
      	    'deltas':[]
      	    }
      	    );
          } else if( this.RelatedEntityId )
          {
      	emit( this.RelatedEntityId, 
      	      {
      		  'entity': undefined,
      		  'deltas': [this]
      	      }
      	    );
          }
      };
      
      
      r = function(key, values){
          /**
           * @description Reducer function
           * @param key
           * @param values Values associated with the given key
           * @return map containing up to 3 fields: deltas with a list of deltas, entity with the entity, key which contains the key
           **/
      
          var reducedPackage = { 'deltas' : [], 
      			   'entity' : undefined };
          for( indexValue in values ){
      	currentValue = values[indexValue];
      	if( currentValue.entity ){
      	    reducedPackage.entity = currentValue.entity;
      	}
      	reducedPackage.deltas = reducedPackage.deltas.concat( currentValue.deltas );
          }
          return reducedPackage;
      };
      

      Additional information:
      =======================
      Avg Object Size: 82 Kb, Size of the database: 60 Mb, Storage: 80 Mb
      Object Count: 801
      Number of MapReduce keys: 1

            Assignee:
            Tad Marshall
            Reporter:
            Mete Dizioglu
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: