Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-52932

Increased operation times after upgrade to 4.4.1

    • Type: Icon: Bug Bug
    • Resolution: Incomplete
    • Priority: Icon: Minor - P4 Minor - P4
    • None
    • Affects Version/s: 4.4.1
    • Component/s: None
    • Labels:
      None
    • Environment:
      Docker containers on Ubuntu 16.04 AWS VMs
    • ALL

      We upgraded 2 replica sets from 4.2.8 to 4.4.1 and saw immediate performance degradation as query times went from milliseconds to seconds

      • The service that uses the upgraded DB (from 4.2.8 to 4.4.1 ) had the P50 response times increase from 3-4ms to 23 seconds. 
      • CPU, memory and disk metrics did on the DB server do not seem to differ 
      • After the upgrade we could not rollback to 4.2 and had to resync from 4.2 members that were not upgraded yet

       

      {"t":\{"$date":"2020-10-01T09:46:25.530+00:00"}

      ,"s":"I","c":"COMMAND","id":51803,"ctx":"conn323349","msg":"Slow query","attr":{"durationMillis":2045}}

      {"t":\{"$date":"2020-10-01T09:46:25.531+00:00"}

      ,"s":"I","c":"COMMAND","id":51803,"ctx":"conn510525","msg":"Slow query","attr":{"durationMillis":8544}}

      {"t":\{"$date":"2020-10-01T09:46:25.531+00:00"}

      ,"s":"I","c":"COMMAND","id":51803,"ctx":"conn506358","msg":"Slow query","attr":{"durationMillis":4049}}

      {"t":\{"$date":"2020-10-01T09:46:25.531+00:00"}

      ,"s":"I","c":"COMMAND","id":51803,"ctx":"conn512184","msg":"Slow query","attr":{"durationMillis":1045}}

      {"t":\{"$date":"2020-10-01T09:46:25.532+00:00"}

      ,"s":"W","c":"COMMAND","id":20525,"ctx":"conn506888","msg":"Failed to gather storage statistics for slow operation","attr":{"durationMillis":null}}

      {"t":\{"$date":"2020-10-01T09:46:25.532+00:00"}

      ,"s":"I","c":"COMMAND","id":51803,"ctx":"conn506888","msg":"Slow query","attr":{"durationMillis":8049}}

      {"t":\{"$date":"2020-10-01T09:46:25.532+00:00"}

      ,"s":"I","c":"COMMAND","id":51803,"ctx":"conn3285","msg":"Slow query","attr":{"durationMillis":1549}}

       I'll attach diagnostic.data from the runs with version 4.4.1 

      The client is Node.js and uses mongodb lib version 3.6.2 

      If you need additional info please let me know. 

        1. Screenshot 2020-11-19 at 17.56.04.png
          Screenshot 2020-11-19 at 17.56.04.png
          110 kB
        2. mongos-mongodb.diagnostic.data.tar.gz
          3.81 MB
        3. mongod-diagnostic.data.tar.gz
          18.45 MB
        4. lock.png
          lock.png
          235 kB
        5. diagnostic.tgz
          25.91 MB
        6. diag.data.tgz
          23.80 MB

            Assignee:
            bruce.lucas@mongodb.com Bruce Lucas (Inactive)
            Reporter:
            alin.silvian@gmail.com Alin Dumitru
            Votes:
            1 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated:
              Resolved: