Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-3092

M/R threshold for running reduces and dumping data to disk are too low, resulting in more CPU and disk I/O

    • Type: Icon: Improvement Improvement
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None

      Right now we:

      • go through whole map every 100 emits, and if it's over 50KB we try to reduce each key
      • every 100 emits, if map is over 100KB, it is dumped to disk in a collection.
        When these threshold are increased you get faster execution.
        Notably a user reported than on a collection with millions of entries and 1KB objects, the incremental dumps actually add up to a massive collection on disk, since there is almost no reduce done ahead of time.

            Assignee:
            antoine Antoine Girbal
            Reporter:
            antoine Antoine Girbal
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - 3 hours
                3h
                Remaining:
                Remaining Estimate - 3 hours
                3h
                Logged:
                Time Spent - Not Specified
                Not Specified