[SERVER-3092] M/R threshold for running reduces and dumping data to disk are too low, resulting in more CPU and disk I/O Created: 13/May/11  Updated: 12/Jul/16  Resolved: 13/May/11

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Antoine Girbal Assignee: Antoine Girbal
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: 3 hours
Time Spent: Not Specified
Original Estimate: 3 hours

Participants:

 Description   

Right now we:

  • go through whole map every 100 emits, and if it's over 50KB we try to reduce each key
  • every 100 emits, if map is over 100KB, it is dumped to disk in a collection.
    When these threshold are increased you get faster execution.
    Notably a user reported than on a collection with millions of entries and 1KB objects, the incremental dumps actually add up to a massive collection on disk, since there is almost no reduce done ahead of time.


 Comments   
Comment by auto [ 13/May/11 ]

Author:

{u'login': u'agirbal', u'name': u'agirbal', u'email': u'antoine@10gen.com'}

Message: SERVER-3092: M/R threshold for running reduces and dumping data to disk are too low, resulting in more CPU and disk I/O
Branch: master
https://github.com/mongodb/mongo/commit/d43037f1982f94a406f2fdcfaaf034fd6ca865b0

Generated at Thu Feb 08 03:02:03 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.