[SERVER-57249] Performance degrade after 4.4.3 up-gradation from 4.2 version Created: 27/May/21  Updated: 01/Jun/22  Resolved: 03/Sep/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.4.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Aayushi Mangal Assignee: Edwin Zhou
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

Hello Team,

We have upgraded our whole cluster to 4.4 and found performance is not better instead degraded.

We have sharded clusters with three replica in each set. We did performance testing by hitting some urls pre and post upgrades.

Please share portal so we can upload logs personally. 

 



 Comments   
Comment by Edwin Zhou [ 03/Sep/21 ]

Hi aayushi.mangal3@gmail.com

We haven’t heard back from you for some time, so I’m going to close this ticket. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Best,
Edwin

Comment by Edwin Zhou [ 17/Aug/21 ]

Hi aayushi.mangal3@gmail.com,

Have you been able to upgrade to MongoDB 4.4 from 4.2? If so, we're interested in:

  • mongod and mongos logs, and the specific timestamps when workload is performed
  • the $dbpath/diagnostic.data (contents described here)

We're looking forward to hearing back from you!

Best,
Edwin

Comment by Edwin Zhou [ 29/Jul/21 ]

Hi aayushi.mangal3@gmail.com,

We're happy to hear that you're preparing to perform another upgrade! The metrics that we primarily need are ones that cover the workload on both 4.2 and 4.4. Can you please provide the following information for those two points?

  • mongod and mongos logs, and the specific timestamps when workload is performed
  • the $dbpath/diagnostic.data (contents described here)

Best,
Edwin

Comment by Aayushi Mangal [ 27/Jul/21 ]

Hi Edwin,

I am able to do upgrade again. But need a week time. Meanwhile please provide me any inputs that you are looking from apart from logs. Like any standard tool we can use before and after upgrade of MongoDB to verify performance. Like keyhole or something?

 

Or any other steps we have to follow while upgrade our shard cluster?

 

We used only jmeter, and found our calls are slow after upgrade. 

 

Comment by Edwin Zhou [ 26/Jul/21 ]

Hi aayushi.mangal3@gmail.com,

We still need additional information to diagnose the problem. If this is still an issue for you, would you please provide your kindly offered jmeter test results so we can understand the severity of this performance regression? If you're able to upgrade again to 4.4, can you provide the mongod logs, mongos logs, and the $dbpath/diagnostic.data covering the workload in 4.2 where the degradation is absent, and for 4.4 where the degradation is present?

Best,
Edwin

Comment by Edwin Zhou [ 12/Jul/21 ]

Hi aayushi.mangal3@gmail.com,

Thank you for following up. Unfortunately, without metrics from the $dbpath/diagnostic.data directory covering workloads between 4.2 and 4.4, it will difficult to determine the root cause of the performance degradation. However, we'd appreciate the jmeter test results which will give us an idea of the severity of the performance degradation. Can you also ensure that timestamps and timezone are visible in the jmeter testing?

Nevertheless, we remain interested in understanding the cause of the performance degradation in 4.4 after upgrading from 4.2. We understand that transitioning between versions is a significant commitment and we will greatly appreciate if you're able to provide additional diagnostics. Can you provide the following information covering the workload in 4.2 where the degradation is absent, and for 4.4 where the degradation is present:

  • mongod and mongos logs, and the specific timestamps when workload is performed
  • the $dbpath/diagnostic.data (contents described here)

Best,
Edwin

Comment by Aayushi Mangal [ 02/Jul/21 ]

Hi Edwin,

 

We dont have specific line showing degradation of performance. We did jmeter testing before and after upgrade. If you would like to I can share that too.

Comment by Edwin Zhou [ 14/Jun/21 ]

Hi aayushi.mangal3@gmail.com,

Thank you for sharing the diagnostic data. Unfortunately it's unclear without a timeline or specific queries what the degradation is based on the data provided. Can you also provide:

  • mongod and mongos logs
  • the mongos $dbpath/diagnostic.data

Can you then provide the specific log lines that suggest the performance degradation after upgrading to 4.4?

Best,
Edwin

Comment by Aayushi Mangal [ 11/Jun/21 ]

Hi Edwin,

Details shared.

Comment by Edwin Zhou [ 04/Jun/21 ]

Hi aayushi.mangal3@gmail.com,

Would you please archive (tar or zip) the $dbpath/diagnostic.data directory (the contents are described here) and upload it using the secure upload portal?

I've created a secure upload portal for you. Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time.

Best,
Edwin

Generated at Thu Feb 08 05:41:22 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.