Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-35958

Big CPU load increase (×4) on secondary by upgrading 3.4.15 → 3.6.5

    XMLWordPrintable

    Details

    • Operating System:
      ALL
    • Steps To Reproduce:
      Hide

      Upgrade RS secondary from 3.4.15 to 3.6.5 and watch CPU usage

      Show
      Upgrade RS secondary from 3.4.15 to 3.6.5 and watch CPU usage
    • Sprint:
      Storage NYC 2019-02-25

      Description

      I just tried upgrading one of my RS from 3.4.15 to 3.6.5, thinking as 4.0 is now released 3.6 might be mature/stable enough now, but I had a very bad surprise, the CPU load increase about 4 times on the secondary (I didn't try the primary of course, I don't want to loose my cluster yet). As you can see on the chart bellow, my mongo usage is very stable and the secondary usually stays at 5% of total server CPU on 3.4.15, as soon as I upgrade to 3.6.5 it jumps to 15-25%. The load stayed exactly the same on the primary, and so does the CPU usage.

       

      When I saw that I thought: maybe I should try re-syncing the secondary to benefit from the improvement of 3.6, so I did this, the first time it filled the disk during the initial sync and crashed mongo so I tried again and the second time it worked, but the load is exactly the same.

      I also tried putting back 3.6.0 and 3.6.1 to see where the regression is, but both didn't boot for unknown reasons yet.

      Is this expected? are you aware of any change that would cause this huge regression?

      This is my mongo cloud manager account monitoring the RS if it helps: https://cloud.mongodb.com/v2/5012a0ac87d1d86fa8c22e64#metrics/replicaSet/5414cfabe4b0ce23e21b4b3b/overview 

        Attachments

        1. comparison.png
          360 kB
          Bruce Lucas
        2. eviction.png
          167 kB
          Bruce Lucas
        3. flush.png
          278 kB
          Bruce Lucas
        4. image-2018-07-04-20-49-54-349.png
          55 kB
          Adrien Jarthon
        5. lag.png
          123 kB
          Bruce Lucas
        6. mongo-3.6-upgrade.png
          58 kB
          Adrien Jarthon
        7. mongodb.log.gz
          1.09 MB
          Adrien Jarthon
        8. mongo-performance.png
          51 kB
          Adrien Jarthon
        9. psa.png
          371 kB
          Bruce Lucas
        10. puzzle.png
          417 kB
          Bruce Lucas
        11. screenshot-1.png
          70 kB
          Adrien Jarthon
        12. screenshot-2.png
          16 kB
          Adrien Jarthon
        13. screenshot-3.png
          52 kB
          Adrien Jarthon
        14. screenshot-4.png
          55 kB
          Adrien Jarthon
        15. screenshot-5.png
          68 kB
          Adrien Jarthon
        16. screenshot-6.png
          25 kB
          Adrien Jarthon
        17. screenshot-7.png
          37 kB
          Adrien Jarthon
        18. screenshot-8.png
          24 kB
          Adrien Jarthon
        19. screenshot-9.png
          21 kB
          Adrien Jarthon
        20. Screenshot from 2018-07-04 08-53-16.png
          29 kB
          Adrien Jarthon
        21. server35958.png
          252 kB
          Alexander Gorrod
        22. timestamps_pinned.png
          121 kB
          Sulabh Mahajan

          Issue Links

            Activity

              People

              Assignee:
              geert.bosch Geert Bosch
              Reporter:
              bigbourin@gmail.com Adrien Jarthon
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              25 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: