-
Type:
Bug
-
Status: Closed
-
Priority:
Major - P3
-
Resolution: Incomplete
-
Affects Version/s: 2.0.0, 2.2.0
-
Fix Version/s: None
-
Component/s: MapReduce
-
Labels:None
-
Operating System:ALL
When running a map/reduce job on a collection containing a large number of entity for a single key in the map/reduce operation, the computation fails because of the size of the object fed into the reduce.
In our case the reduce accumulates the values encountered in an array. The finalize would after trim down these values and apply the final operations to the final object. This accumulation process cannot be filtered before and needs all the informations.
Code:
=====
m = function(){
|
/**
|
* @description Mapper
|
* @return
|
**/
|
if( this._attributes ){
|
emit(
|
this._attributes.entityid,
|
{
|
'entity': this,
|
'deltas':[]
|
}
|
);
|
} else if( this.RelatedEntityId )
|
{
|
emit( this.RelatedEntityId,
|
{
|
'entity': undefined,
|
'deltas': [this]
|
}
|
);
|
}
|
};
|
|
|
|
|
r = function(key, values){
|
/**
|
* @description Reducer function
|
* @param key
|
* @param values Values associated with the given key
|
* @return map containing up to 3 fields: deltas with a list of deltas, entity with the entity, key which contains the key
|
**/
|
|
|
var reducedPackage = { 'deltas' : [],
|
'entity' : undefined };
|
for( indexValue in values ){
|
currentValue = values[indexValue];
|
if( currentValue.entity ){
|
reducedPackage.entity = currentValue.entity;
|
}
|
reducedPackage.deltas = reducedPackage.deltas.concat( currentValue.deltas );
|
}
|
return reducedPackage;
|
};
|
Additional information:
=======================
Avg Object Size: 82 Kb, Size of the database: 60 Mb, Storage: 80 Mb
Object Count: 801
Number of MapReduce keys: 1