Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-71503

Big disk IO (write) regression (×8) when setting fCV to 4.4

    • Type: Icon: Bug Bug
    • Resolution: Gone away
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 4.4.18
    • Component/s: None
    • Labels:
      None
    • Replication
    • ALL

      I just tried upgrading my RS running 4.4.18 from fCV 4.2 to fCV 4.4 (yes it has been running as 4.2 for a white ^^). Unfortunately bad surprise, I immediately saw the disk IO load (write) on both secondary jump to the roof (about 8 times from ~12% to ~90%). The amount of data written only slightly increased though, so it's mostly the number of writes, which are now smaller I suppose. There's also a ×2 increase in the CPU usage and a small network usage increase, probably related. As soon as I downgraded back to fCV 4.2, the IO, CPU and network load went down back to previous levels (times are UTC+1 here):

      CPU:

      I checked of course the features which requires fCV 4.4 but there's only two listed here and I don't see any likely culprit: https://www.mongodb.com/docs/manual/release-notes/4.4-compatibility/#std-label-4.4-compatibility-enabled

      So I suppose there are other changes caused by this flag, are you aware of any change that could cause this huge regression? Maybe there's a configuration default which changes that I could tune?

      I have the diagnostic.data and logs for these servers if you let me know where I can upload them safely.

      I wanted to do this in order to prepare the upgrade to 5.0 and 6.0, but I can't really continue with the upgrade or try newer versions if 4.4 already puts me in such a difficult position. Also sorry if there's already a ticket about this, I did try to search but couldn't find any.

      Thanks!

        1. image-2022-11-18-23-37-51-756.png
          image-2022-11-18-23-37-51-756.png
          91 kB
        2. image-2022-11-18-23-40-32-446.png
          image-2022-11-18-23-40-32-446.png
          25 kB
        3. image-2022-11-18-23-54-38-636.png
          image-2022-11-18-23-54-38-636.png
          70 kB
        4. image-2022-11-22-11-03-08-057.png
          image-2022-11-22-11-03-08-057.png
          29 kB
        5. image-2022-12-21-16-29-47-844.png
          image-2022-12-21-16-29-47-844.png
          36 kB
        6. mongodb-4.4-secondary-IO-regression.png
          mongodb-4.4-secondary-IO-regression.png
          108 kB
        7. SERVER-71503-diskIOPS.png
          SERVER-71503-diskIOPS.png
          547 kB
        8. SERVER-71503-getMores-increse.png
          SERVER-71503-getMores-increse.png
          524 kB

            Assignee:
            backlog-server-repl [DO NOT USE] Backlog - Replication Team
            Reporter:
            bigbourin@gmail.com Adrien Jarthon
            Votes:
            0 Vote for this issue
            Watchers:
            14 Start watching this issue

              Created:
              Updated:
              Resolved: