Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-17010

Reduce file handle usage in File based Sorter

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major - P3
    • Resolution: Done
    • 2.6.7, 3.2.1
    • 3.4.22, 3.6.12, 4.0.7, 4.1.5
    • Storage
    • Fully Compatible
    • ALL
    • v4.0, v3.6, v3.4
    • Storage NYC 2018-09-24, Storage NYC 2018-10-08, Storage NYC 2018-10-22, Storage NYC 2018-11-05
    • 90

    Description

      Per dupuisla, we need to reduce the number of file handles that we used to sort large amounts of data to avoid file handle exhaustion on various platforms.

      See SERVER-14572:

      I spent some time on the sorter.cpp and it is clear that you need to review this code. The FileIterator consume far too much file descriptors. There is no upper limit and increase the number of handle is just a poor fix.
      Why not merge all this in one file and use "seek" to move between the different block instead of creating thousand of files? In my case, I have something like 2000 of these temp files created, so 2048 will be on the low side.

      Attachments

        Issue Links

          has to be done before

          Improvement - An improvement or enhancement to an existing feature or task. SERVER-37293 Refactor Sorter so that DocumentSourceGroup can use it instead of SortedFileWriter implementation

          • Major - P3 - Major loss of function.
          • Closed
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. SERVER-39597 Reduce file handle usage in File based Sorter

          • Critical - P2 - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. SERVER-31652 Initialization synchronization of windows 2012 R2 mongodb replica set errno:24 Too many open files

          • Major - P3 - Major loss of function.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. SERVER-38764 External sorter should use 64-bit integers for file offsets

          • Major - P3 - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. SERVER-24020 Increase open file limit on Windows

          • Major - P3 - Major loss of function.
          • Closed
          related to

          Bug - A problem which impairs or prevents the functions of the product. SERVER-16991 errno:24 Too many open files on Windows

          • Major - P3 - Major loss of function.
          • Closed

          Activity

            People

              dianna.hohensee@mongodb.com Dianna Hohensee (Inactive)
              mark.benvenuto@mongodb.com Mark Benvenuto
              Votes:
              7 Vote for this issue
              Watchers:
              25 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: