Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-7037

Error in M/R -- 'value too large to reduce'

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Incomplete
    • Affects Version/s: 2.0.0, 2.2.0
    • Fix Version/s: None
    • Component/s: MapReduce
    • Labels:
      None
    • Operating System:
      ALL

      Description

      When running a map/reduce job on a collection containing a large number of entity for a single key in the map/reduce operation, the computation fails because of the size of the object fed into the reduce.
      In our case the reduce accumulates the values encountered in an array. The finalize would after trim down these values and apply the final operations to the final object. This accumulation process cannot be filtered before and needs all the informations.

      Code:
      =====

      m = function(){
          /**
           * @description Mapper
           * @return 
           **/
          if( this._attributes ){
      	emit(
      	    this._attributes.entityid, 
      	    {
      	    'entity': this,
      	    'deltas':[]
      	    }
      	    );
          } else if( this.RelatedEntityId )
          {
      	emit( this.RelatedEntityId, 
      	      {
      		  'entity': undefined,
      		  'deltas': [this]
      	      }
      	    );
          }
      };
       
       
      r = function(key, values){
          /**
           * @description Reducer function
           * @param key
           * @param values Values associated with the given key
           * @return map containing up to 3 fields: deltas with a list of deltas, entity with the entity, key which contains the key
           **/
       
          var reducedPackage = { 'deltas' : [], 
      			   'entity' : undefined };
          for( indexValue in values ){
      	currentValue = values[indexValue];
      	if( currentValue.entity ){
      	    reducedPackage.entity = currentValue.entity;
      	}
      	reducedPackage.deltas = reducedPackage.deltas.concat( currentValue.deltas );
          }
          return reducedPackage;
      };

      Additional information:
      =======================
      Avg Object Size: 82 Kb, Size of the database: 60 Mb, Storage: 80 Mb
      Object Count: 801
      Number of MapReduce keys: 1

        Attachments

          Activity

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: