Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-35958

Big CPU load increase (×4) on secondary by upgrading 3.4.15 → 3.6.5

    • ALL
    • Hide

      Upgrade RS secondary from 3.4.15 to 3.6.5 and watch CPU usage

      Show
      Upgrade RS secondary from 3.4.15 to 3.6.5 and watch CPU usage
    • Storage NYC 2019-02-25

      I just tried upgrading one of my RS from 3.4.15 to 3.6.5, thinking as 4.0 is now released 3.6 might be mature/stable enough now, but I had a very bad surprise, the CPU load increase about 4 times on the secondary (I didn't try the primary of course, I don't want to loose my cluster yet). As you can see on the chart bellow, my mongo usage is very stable and the secondary usually stays at 5% of total server CPU on 3.4.15, as soon as I upgrade to 3.6.5 it jumps to 15-25%. The load stayed exactly the same on the primary, and so does the CPU usage.

       

      When I saw that I thought: maybe I should try re-syncing the secondary to benefit from the improvement of 3.6, so I did this, the first time it filled the disk during the initial sync and crashed mongo so I tried again and the second time it worked, but the load is exactly the same.

      I also tried putting back 3.6.0 and 3.6.1 to see where the regression is, but both didn't boot for unknown reasons yet.

      Is this expected? are you aware of any change that would cause this huge regression?

      This is my mongo cloud manager account monitoring the RS if it helps: https://cloud.mongodb.com/v2/5012a0ac87d1d86fa8c22e64#metrics/replicaSet/5414cfabe4b0ce23e21b4b3b/overview 

        1. mongo-3.6-upgrade.png
          mongo-3.6-upgrade.png
          58 kB
        2. Screenshot from 2018-07-04 08-53-16.png
          Screenshot from 2018-07-04 08-53-16.png
          29 kB
        3. image-2018-07-04-20-49-54-349.png
          image-2018-07-04-20-49-54-349.png
          55 kB
        4. screenshot-1.png
          screenshot-1.png
          70 kB
        5. lag.png
          lag.png
          123 kB
        6. eviction.png
          eviction.png
          167 kB
        7. timestamps_pinned.png
          timestamps_pinned.png
          121 kB
        8. mongo-performance.png
          mongo-performance.png
          51 kB
        9. screenshot-2.png
          screenshot-2.png
          16 kB
        10. psa.png
          psa.png
          371 kB
        11. screenshot-3.png
          screenshot-3.png
          52 kB
        12. mongodb.log.gz
          1.09 MB
        13. screenshot-4.png
          screenshot-4.png
          55 kB
        14. screenshot-5.png
          screenshot-5.png
          68 kB
        15. server35958.png
          server35958.png
          252 kB
        16. comparison.png
          comparison.png
          360 kB
        17. puzzle.png
          puzzle.png
          417 kB
        18. screenshot-6.png
          screenshot-6.png
          25 kB
        19. screenshot-7.png
          screenshot-7.png
          37 kB
        20. screenshot-8.png
          screenshot-8.png
          24 kB
        21. flush.png
          flush.png
          278 kB
        22. screenshot-9.png
          screenshot-9.png
          21 kB

            Assignee:
            geert.bosch@mongodb.com Geert Bosch
            Reporter:
            bigbourin@gmail.com Adrien Jarthon
            Votes:
            0 Vote for this issue
            Watchers:
            24 Start watching this issue

              Created:
              Updated:
              Resolved: