[SERVER-85040] Measure sys-perf performance with SBE Created: 23/Apr/20  Updated: 12/Jan/24  Resolved: 06/May/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: David Storch Assignee: Anton Korshunov
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Sprint: Query 2020-05-04, Query 2020-05-18
Participants:

 Comments   
Comment by Anton Korshunov [ 07/May/20 ]

david.storch, martin.neupauer OK, I've got the correct numbers. Overall, we're better than MDB for most workloads! With a few exceptions though, see below.

  1. Query.
    • Improvement is between 5-57%.
    • There are two pretty bad regressions though at 85% for a marginal case when we use a projection consisting of 131 fields. This is a known problem which we discussed with Martin some time ago. Basically, it's a combination of two unfortunate factors that we call getField on each projected field, and then use the mkobj stage which is not super efficient. We were thinking to change the latter to construct a BSONObj on the fly, rather than an SBE Object and then convert into a BSONObj in the PlanExecutor, but we never did it.
  2. Aggregate.
    • Improvement is modest at 2-20%.
    • Same 85% regression with large projections.
    • And another two workloads with a projection show 23-30% regression. The same workloads using Query show improvement (one at 52%). I don't know how to explain it though.

Here are the results compared against two previous runs: query and agg.

Comment by Anton Korshunov [ 06/May/20 ]

I probably could, but then I'd have to rerun the compile task again. It's running now with SBE turned on.

Comment by David Storch [ 06/May/20 ]

Can you just change query_knobs.idl to turn SBE on by default instead of patching dsi?

Comment by Anton Korshunov [ 06/May/20 ]

Wait, what?! Did I really forget to patch a dsi module? Oh boy. Alright, bear with me, I will rerun it shortly. Thanks for the great catch!

Comment by David Storch [ 06/May/20 ]

Thanks anton.korshunov! Are you sure the patch build you ran is measuring SBE performance, though? I notice that the patch submitted to Evergreen did not change the default value of internalQueryEnableSlotBasedExecutionEngine to true. Am I missing something, or was this a mistake? Assuming that these numbers are really from SBE execution, do you have any intuition as to why the find queries show no improvement? These seem like the kind of queries that might show a real benefit from SBE, even if the improvement is small.

Comment by Anton Korshunov [ 05/May/20 ]

I executed the sys-perf benchmark, and the results are good overall. The only relevant workload was bestbuy_agg_query_comparison, which can be run using either find or aggregate engines. So, I did both and the results can be found here.

Notably, there was no regression in the aggregate version of the workload. But there was no any significant improvement either.

The find version was also mostly fine, with the only test showing 14% regression being the find_limit test. I'm pretty sure this is due to the fact that we combined the skip and limit stages into a single limitskip stage, and we use a boost::optional on each getNext call to compare the current counter agains the limit. We may want to consider splitting the limitskip stage into two stages to buy back the performance.

david.storch martin.neupauer, FYI.

Comment by Anton Korshunov [ 30/Apr/20 ]

Sysperf workloads are failing on getMore somewhere deep in WT. I'm going to give it another try once we rebase to latest master.

Generated at Thu Feb 08 06:56:39 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.