[SERVER-71542] Investigate changepoints in bestbuy_4_analytics Created: 22/Nov/22 Updated: 29/Oct/23 Resolved: 29/Nov/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 6.3.0-rc0 |
| Type: | Task | Priority: | Major - P3 |
| Reporter: | Steve Tarzia | Assignee: | Steve Tarzia |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | pm2646-m4 | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||
| Backwards Compatibility: | Fully Compatible | ||||
| Sprint: | QE 2022-11-28, QE 2022-12-12 | ||||
| Participants: | |||||
| Description |
|
Performance improved drastically (up to 50,000x) on several tests in bestbuy_4_analytics due to a change sometime between Nov 16th and 18th. Most likely causes are: The results have been reproduced several times in evergreen, as you can see in the trend charts here: https://spruce.mongodb.com/version/637bb59cc9ec4412eb36da6a/tasks?sorts=STATUS%3AASC%3BBASE_STATUS%3ADESC Query MQL is listed here: https://github.com/10gen/workloads/blob/master/workloads/bestbuy_analytics.js |
| Comments |
| Comment by Steve Tarzia [ 22/Nov/22 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I found the issue. A recent DSI change removed the --noIndexRestore from test_control.bestbuy_analytics.yml. We have several test control files for bestbuy and someone copy-pasted the same commands to all of them, but that's not appropriate because some tests have indexes restored and others do not: I'll put up a simple DSI patch to revert this change. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Steve Tarzia [ 22/Nov/22 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
ian.boros@mongodb.com good point! Now that I look at my test_control.bestbuy_analytics.yml I see a comment claiming that we don't restore indexes, but the actual mongorestore command is missing the desired --noIndexRestore. I'll investigate this and check the local behavior with indexes. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Ian Boros [ 22/Nov/22 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The error theory sounds plausible. One other thought: Is it at all possible an index is now being used "by accident"? I realize the test doesn't create indexes other than the CSI, but do the indexes created as part of the `mongorestore` stick around? | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Steve Tarzia [ 22/Nov/22 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
When I run the 1_col_nomatch_scalar query locally (on a scale-1 bestbuy collection), I get a runtime of about 800ms. This involves a full collection scan. There is no conceivable way that the query running in Evergreen should be seeing 1600 operations per second. It previously was getting 0.03 ops per second in Evergreen. My assumption at this point is that the query is somehow silently failing. I'll add an assertion to the workload and run again in evergreen. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Steve Tarzia [ 22/Nov/22 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Here are a couple of sample queries that changed: 1_col_nomatch_scalar in bestbuy_4_analytics showing a 50,000x speedup:
match_3_column in bestbuy_4_analytics showing a 100x speedup:
|