Uploaded image for project: 'MongoDB ETL Tools'
  1. MongoDB ETL Tools
  2. TOOLS-1613

mongodump Ensure that all disk writes are buffered

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Accepted
    • Priority: Major - P3
    • Resolution: Unresolved
    • Affects Version/s: 3.4.2
    • Fix Version/s: None
    • Component/s: mongodump
    • Environment:
      Windows 7 64bit

      Description

      When performing a mongodump of a large database I encountered several times the problem that mongodump increases the NTFS drive fragmentation beyond the level the OS can handle. Hence in my case the dump process always fails at a certain point.

      Using ProcessMonitor I investigated how mongodump writes the data to disk this what I found using the following commands:

      mongodump.exe /gzip /db:testdb /out:D:\DumpTest
      mongodump.exe /db:testdb /out:D:\DumpTest
      

      Using the first command with /gzip mongodump writes the to disk in chunks of 240 bytes!
      Using the second command without /gzip the size of written data varies randomly between 4KB and about 260KB.

      Conclusion: mongodump does not use an application internal write buffer (or if it uses one not of an appropriate size). For a tool like mongoump I would have expected a write buffer of at least one Megabyte per thread. This would drastically improve the write speed especially considering modern storage system like flash disks which are internally not organized in blocks of 512 bytes or 4096 bytes.

      Therefore please add a disk write buffer for all output operations of mongodump. The best way would be to make the buffer size user configurable.

        Attachments

          Activity

            People

            • Assignee:
              backlog-server-tools Backlog - Server Tools Team
              Reporter:
              mango Jan S.
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: