[SERVER-28172] Renaming collection causes permanent performance issue on secondary Created: 02/Mar/17 Updated: 31/May/17 Resolved: 05/May/17 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 3.4.2 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Marc Tinkler | Assignee: | Mark Agarunov |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||
| Issue Links: |
|
||||||||||||||||
| Operating System: | ALL | ||||||||||||||||
| Steps To Reproduce: | Here's the rough pseudocode to reproduce:
|
||||||||||||||||
| Participants: | |||||||||||||||||
| Description |
| Comments |
| Comment by Marc Tinkler [ 05/May/17 ] | ||||||
|
Thomas, We've upgraded and everything is working again. Thanks! | ||||||
| Comment by Kelsey Schubert [ 04/May/17 ] | ||||||
|
We've also backported Thank you, | ||||||
| Comment by Mark Agarunov [ 29/Mar/17 ] | ||||||
|
Hello tinkler@vocabulary.com, Thank you for providing the files. Looking over the contents, I believe the behavior you're seeing may be related to the issue described in Thanks, | ||||||
| Comment by Marc Tinkler [ 03/Mar/17 ] | ||||||
|
Thanks for the clarification Mark, I uploaded those logs. | ||||||
| Comment by Mark Agarunov [ 03/Mar/17 ] | ||||||
|
Hello tinkler@vocabulary.com, Thank you for providing the log. I've created a secure upload portal for you to use. However please note that the diagnostic.data directory does not contain any of your data. It periodically collects the output of the following commands, which you are welcome to execute yourself to examine the output:
Thanks, | ||||||
| Comment by Marc Tinkler [ 03/Mar/17 ] | ||||||
|
Here is the portion of the log from right before the problem happened until reboot. I can also provide the diagnostic.data, but it's over 150mb gzipped and I'd prefer to provide it to you securely. Please let me know how you'd like me to do that. | ||||||
| Comment by Marc Tinkler [ 03/Mar/17 ] | ||||||
|
Here's another screen shot of MMS after it happened again. I will get you logs for this period and attach them to this issue. This is connections and Operation execution times | ||||||
| Comment by Marc Tinkler [ 03/Mar/17 ] | ||||||
|
Hey Mark, That's not exactly it. I don't know if you can reproduce these conditions in house, but under these circumstances, we can very reliably reproduce it. We are happy to host a screen share with you so you can see it happen in real time. Here's what you need to do:
I will get you the logs and diagnostics ASAP. | ||||||
| Comment by Mark Agarunov [ 02/Mar/17 ] | ||||||
|
Hello tinkler@vocabulary.com, Thank you for the report. Unfortunately, we're having difficulty reproducing the behavior you are describing. To attempt to reproduce, I followed the following steps, please let me know if something was missed:
Additionally, to get a better idea of what may be causing the issue, please provide the following:
Thanks, | ||||||
| Comment by Marc Tinkler [ 02/Mar/17 ] | ||||||
|
Here's another tidbit of information: it seems we are able to reproduce the same problem by simply dropping a collection. But it's hit-or-miss, sometime it happens, and sometimes not. Also, it does not seem to be related to the size of the collection we are dropping. We were able to reproduce it by dropping a collection with a single document. |