Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-52932

Increased operation times after upgrade to 4.4.1

    • Type: Icon: Bug Bug
    • Resolution: Incomplete
    • Priority: Icon: Minor - P4 Minor - P4
    • None
    • Affects Version/s: 4.4.1
    • Component/s: None
    • Labels:
      None
    • Environment:
      Docker containers on Ubuntu 16.04 AWS VMs
    • ALL

      We upgraded 2 replica sets from 4.2.8 to 4.4.1 and saw immediate performance degradation as query times went from milliseconds to seconds

      • The service that uses the upgraded DB (from 4.2.8 to 4.4.1 ) had the P50 response times increase from 3-4ms to 23 seconds. 
      • CPU, memory and disk metrics did on the DB server do not seem to differ 
      • After the upgrade we could not rollback to 4.2 and had to resync from 4.2 members that were not upgraded yet

       

      {"t":\{"$date":"2020-10-01T09:46:25.530+00:00"}

      ,"s":"I","c":"COMMAND","id":51803,"ctx":"conn323349","msg":"Slow query","attr":{"durationMillis":2045}}

      {"t":\{"$date":"2020-10-01T09:46:25.531+00:00"}

      ,"s":"I","c":"COMMAND","id":51803,"ctx":"conn510525","msg":"Slow query","attr":{"durationMillis":8544}}

      {"t":\{"$date":"2020-10-01T09:46:25.531+00:00"}

      ,"s":"I","c":"COMMAND","id":51803,"ctx":"conn506358","msg":"Slow query","attr":{"durationMillis":4049}}

      {"t":\{"$date":"2020-10-01T09:46:25.531+00:00"}

      ,"s":"I","c":"COMMAND","id":51803,"ctx":"conn512184","msg":"Slow query","attr":{"durationMillis":1045}}

      {"t":\{"$date":"2020-10-01T09:46:25.532+00:00"}

      ,"s":"W","c":"COMMAND","id":20525,"ctx":"conn506888","msg":"Failed to gather storage statistics for slow operation","attr":{"durationMillis":null}}

      {"t":\{"$date":"2020-10-01T09:46:25.532+00:00"}

      ,"s":"I","c":"COMMAND","id":51803,"ctx":"conn506888","msg":"Slow query","attr":{"durationMillis":8049}}

      {"t":\{"$date":"2020-10-01T09:46:25.532+00:00"}

      ,"s":"I","c":"COMMAND","id":51803,"ctx":"conn3285","msg":"Slow query","attr":{"durationMillis":1549}}

       I'll attach diagnostic.data from the runs with version 4.4.1 

      The client is Node.js and uses mongodb lib version 3.6.2 

      If you need additional info please let me know. 

        1. diag.data.tgz
          23.80 MB
        2. diagnostic.tgz
          25.91 MB
        3. lock.png
          lock.png
          235 kB
        4. mongod-diagnostic.data.tar.gz
          18.45 MB
        5. mongos-mongodb.diagnostic.data.tar.gz
          3.81 MB
        6. Screenshot 2020-11-19 at 17.56.04.png
          Screenshot 2020-11-19 at 17.56.04.png
          110 kB

            Assignee:
            bruce.lucas@mongodb.com Bruce Lucas (Inactive)
            Reporter:
            alin.silvian@gmail.com Alin Dumitru
            Votes:
            1 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated:
              Resolved: