Uploaded image for project: 'PHP Legacy Driver'
  1. PHP Legacy Driver
  2. PHP-1461

High memory usage during GridFS::storeFile() execution

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor - P4
    • Resolution: Won't Fix
    • Affects Version/s: 1.6.9
    • Fix Version/s: 1.6.x
    • Component/s: pecl-mongo
    • Labels:
      None
    • # Replies:
      1
    • Last comment by Customer:
      true

      Description

      Configuration

      For our tests, we used random input data created cat /dev/urandom > random.txt.

      Set the URI, database, and GridFS collection names accordingly. Additionally, decide if the GridFS collection should be dropped before inserting any data.

      The number of iterations can be customized. Based on the results, there were modest increases to peak memory on successive iterations. Real memory usage tended to drop with each new iteration and climb again.

      By default, the driver uses a chunk size of 255 * 1024 (i.e. 255K). This default seems to leak memory (at least until the file insert completes). Increasing the chunk size even just a bit 256K has vastly better results.

      The logging of actual memory usage at each insert operation can be toggled. It's fine to leave this disabled, as we're primarily concerned with peak usage after running all iterations; however, it can be useful to see the rate of increase (particularly with smaller chunk sizes).

      Lastly, we can configure whether to pass a PHP stream (from fopen()) or a filename to MongoGridFS::storeFile(). In testing, toggling this option did not cause results to significantly differ.

      Results

      The following tests were run with three iterations and a 357M input file. Inserts were not logged and all tests used streams. The driver and server versions were 1.6.9 and 3.0.4, respectively.

      Chunk size: 255K

      Iteration: 0
      Inserted file 55881b86e84df16c6e8b4567 with size: 356.13MB 
      Peak Memory / PHP: 2.34MB / Real: 217.75MB
      ================================================================================
      Iteration: 1
      Inserted file 55881b89e84df16c6e8b4aff with size: 356.13MB 
      Peak Memory / PHP: 2.38MB / Real: 230.00MB
      ================================================================================
      Iteration: 2
      Inserted file 55881b8ce84df16c6e8b5097 with size: 356.13MB 
      Peak Memory / PHP: 2.40MB / Real: 230.00MB
      

      Chunk size: 256K

      Iteration: 0
      Inserted file 55881bebe84df1707d8b4567 with size: 356.13MB 
      Peak Memory / PHP: 2.27MB / Real: 3.25MB
      ================================================================================
      Iteration: 1
      Inserted file 55881befe84df1707d8b4af9 with size: 356.13MB 
      Peak Memory / PHP: 2.28MB / Real: 3.25MB
      ================================================================================
      Iteration: 2
      Inserted file 55881bf2e84df1707d8b508b with size: 356.13MB 
      Peak Memory / PHP: 2.30MB / Real: 3.25MB
      

      Chunk size: 1024K

      Iteration: 0
      Inserted file 55881c06e84df18c028b4567 with size: 356.13MB 
      Peak Memory / PHP: 3.58MB / Real: 4.50MB
      ================================================================================
      Iteration: 1
      Inserted file 55881c10e84df18c028b46cd with size: 356.13MB 
      Peak Memory / PHP: 3.59MB / Real: 4.50MB
      ================================================================================
      Iteration: 2
      Inserted file 55881c14e84df18c028b4833 with size: 356.13MB 
      Peak Memory / PHP: 3.59MB / Real: 4.50MB
      

        Attachments

          Activity

            People

            Assignee:
            Unassigned
            Reporter:
            jmikola Jeremy Mikola
            Participants:
            Last commenter:
            Rathi Gnanasekaran
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Days since reply:
              4 years, 10 weeks, 4 days ago
              Date of 1st Reply: