for inline mapreduce, all emitted objects are kept in RAM before the 1st reduce, potential high memory usage

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Done
    • Priority: Major - P3
    • 1.7.5
    • Affects Version/s: None
    • Component/s: None
    • None
    • ALL
    • None
    • 3
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      during map phase, checkSize() is called to do reduceInMemory and potentially dumpToInc.
      But if inline mode, checkSize doesnt do anything.
      All objects will be emitted before 1st attempt to reduce.
      Instead reduceInMemory should be called if map is over a certain size, or if there is potential for reduce.

            Assignee:
            Antoine Girbal (Inactive)
            Reporter:
            Antoine Girbal (Inactive)
            Votes:
            1 Vote for this issue
            Watchers:
            0 Start watching this issue

              Created:
              Updated:
              Resolved: