Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-17010

Reduce file handle usage in File based Sorter

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: 2.6.7, 3.2.1
    • Fix Version/s: 3.4.22, 3.6.12, 4.0.7, 4.1.5
    • Component/s: Storage
    • Labels:
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Backport Requested:
      v4.0, v3.6, v3.4
    • Sprint:
      Storage NYC 2018-09-24, Storage NYC 2018-10-08, Storage NYC 2018-10-22, Storage NYC 2018-11-05
    • Case:
    • Linked BF Score:
      90

      Description

      Per Laurent Dupuis, we need to reduce the number of file handles that we used to sort large amounts of data to avoid file handle exhaustion on various platforms.

      See SERVER-14572:

      I spent some time on the sorter.cpp and it is clear that you need to review this code. The FileIterator consume far too much file descriptors. There is no upper limit and increase the number of handle is just a poor fix.
      Why not merge all this in one file and use "seek" to move between the different block instead of creating thousand of files? In my case, I have something like 2000 of these temp files created, so 2048 will be on the low side.

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                7 Vote for this issue
                Watchers:
                26 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: