[SERVER-68245] Degraded performance after 4.4 version upgrade Created: 23/Jul/22 Updated: 18/Oct/22 Resolved: 18/Oct/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 4.4.15 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Ishrat Jahan | Assignee: | Chris Kelly |
| Resolution: | Done | Votes: | 0 |
| Labels: | performance | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Operating System: | ALL |
| Participants: |
| Description |
|
Hi, We metrics.2022-07-22T00-20-13Z-00000 The version upgrade to 4.4 was done around 21st July 10:00 pm. Immediately after the upgrade we did not notice any issues, but after a few hours, we started seeing an increase in response time and a lot of slow queries on mongo side. Data size: 5TB(uncompressed) We also observed the following:
Please find attached the screenshots and the diagnostic data. Please help us in finding out the issue and suggest an appropriate fix for this.
|
| Comments |
| Comment by Chris Kelly [ 18/Oct/22 ] |
|
We haven’t heard back from you for some time, so I’m going to close this ticket. If this is still an issue for you, please provide additional information and we will reopen the ticket. |
| Comment by Chris Kelly [ 04/Oct/22 ] |
|
Tejas, We still need additional information to diagnose the problem. If this is still an issue for you, would you please supply the requested information? Specifically, it is difficult to ascertain what degradation you're seeing without an example:
Christopher |
| Comment by Chris Kelly [ 12/Sep/22 ] |
|
Hi Tejas, Thank you for your patience on this. In order to investigate this further, it is imperative we get something that is reproducible. If you are unable to downgrade to 4.2 to collect diagnostic data for comparison, it is at least necessary to provide a workload that is known to be cause this problem, such as a test workload that can be repeated on 4.2 and 4.4. As Eric mentioned, because of the differences in 4.2 and 4.4, asking our community for help by posting on the MongoDB Developer Community Forums may prove helpful for tuning your performance. Regards, |
| Comment by Tejas Jadhav [ 18/Aug/22 ] |
|
Hello Eric, We are not able to repeat this on other workload since this is production instance and it would be hard to replicate the traffic as well as the setup. Nonetheless, we are still seeing the response time spikes everyday. We also won't be able to provide any diagnostic data for 4.2 since that would require a downgrade. Is there anything else we can provide on the current setup? Also, all times in the above screenshots are in IST (+05:30). |
| Comment by Eric Sedor [ 12/Aug/22 ] |
|
Hi ishrat.jahan@gojek.com, and apologies for the delay addressing this report. I want to start by setting some expectations:
To have the best chance of implicating a cause of a performance regression like this, we would really need to see:
Are you able to provide this? Finally, and because of the differences in 4.2 and 4.4, I might also suggest treating this as a performance tuning effort on 4.4 and asking our community for help by posting on the MongoDB Developer Community Forums. They might be able to help zero in on specific improvements to make. Sincerely, |
| Comment by Ishrat Jahan [ 01/Aug/22 ] |
|
Our p99 response time has spiked till 22ms. Can someone please help check this ? |