-
Type: Improvement
-
Resolution: Done
-
Priority: Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
Right now we:
- go through whole map every 100 emits, and if it's over 50KB we try to reduce each key
- every 100 emits, if map is over 100KB, it is dumped to disk in a collection.
When these threshold are increased you get faster execution.
Notably a user reported than on a collection with millions of entries and 1KB objects, the incremental dumps actually add up to a massive collection on disk, since there is almost no reduce done ahead of time.