Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-30477

Mongodump very slow on 3.4.6

    • Type: Icon: Bug Bug
    • Resolution: Gone away
    • Priority: Icon: Minor - P4 Minor - P4
    • None
    • Affects Version/s: 3.4.6
    • Component/s: Performance, Tools
    • Labels:
      None
    • Environment:
      Ubuntu 16.04
      Hosted on Azure

      We have had a problem with mongodump, but believe it to be fixed/worked-around for now. I am sharing this for you guys at mongo to analyze and potentially other users in the same situation to maybe solve their poblem. If this is not the right place for this, please let me know.

      We are using mongodump to make backups of our data, the mongodump is run on a hidden secondary and during night so as not to impact production environment. Before (on 3.2 and 3.4.5) we have not has any difficulties with this. On 3.4.5 the mongodump would run for about 2-3 min on our 1.5 GB databases. After upgrading to 3.4.6, we saw an increase in time each night the dump was run: first 3 min (as the previous version), 9 min, then 46 min, then over 3 hours. For the last one, the mongod instance also apparently crashed and restarted. We are using cloud mongo to monitor our mongod instances and the instance, on which the mongodump was run, was reported to be down for about 2 hours while the mongodump was running. Furthermore, other instances on mongod and the monitor agent on the same server were reported down/up sporadically while the mongodump was running.

      Investigating the disk usage shows that after upgrading to 3.4.6, the disk read has been increasing similarly to the run time. 4 GB/h, 86 GB/h, 94 GB/h, ~110 GB/h for 3+ hours.

      We ran an unattended-upgrade on the server hosting the mongo instance and after that the following mongodump took only 3 min and the disk read was back at 4 GB/h. I don't know if the upgrade fixed it or if it was just coincidental and sporadic. I am not too keen on trying to revert the upgrades to reproduce this on our production server. The packages upgraded were:
      linux-cloud-tools-common linux-cloud-tools-virtual linux-headers-generic linux-headers-virtual linux-image-extra-virtual linux-image-generic linux-image-virtual linux-virtual
      Packages auto removed:
      linux-cloud-tools-4.4.0-83 linux-cloud-tools-4.4.0-83-generic

      If you need any more info, let me know and I try to provide it.

        1. ftdc.png
          ftdc.png
          104 kB
        2. screenshot-1.png
          screenshot-1.png
          38 kB

            Assignee:
            mark.agarunov Mark Agarunov
            Reporter:
            aba@medei.dk Alex Bondo Andersen
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: