Uploaded image for project: 'MongoDB Database Tools'
  1. MongoDB Database Tools
  2. TOOLS-1613

mongodump Ensure that all disk writes are buffered

    • Type: Icon: Task Task
    • Resolution: Unresolved
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.4.2
    • Component/s: mongodump
    • Environment:
      Windows 7 64bit

      When performing a mongodump of a large database I encountered several times the problem that mongodump increases the NTFS drive fragmentation beyond the level the OS can handle. Hence in my case the dump process always fails at a certain point.

      Using ProcessMonitor I investigated how mongodump writes the data to disk this what I found using the following commands:

      mongodump.exe /gzip /db:testdb /out:D:\DumpTest
      mongodump.exe /db:testdb /out:D:\DumpTest
      

      Using the first command with /gzip mongodump writes the to disk in chunks of 240 bytes!
      Using the second command without /gzip the size of written data varies randomly between 4KB and about 260KB.

      Conclusion: mongodump does not use an application internal write buffer (or if it uses one not of an appropriate size). For a tool like mongoump I would have expected a write buffer of at least one Megabyte per thread. This would drastically improve the write speed especially considering modern storage system like flash disks which are internally not organized in blocks of 512 bytes or 4096 bytes.

      Therefore please add a disk write buffer for all output operations of mongodump. The best way would be to make the buffer size user configurable.

            Assignee:
            Unassigned Unassigned
            Reporter:
            mango Jan S.
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: