-
Type:
Improvement
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
DevProd Build
-
200
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Problem
$percentile / $median aggregation hot paths regress ~16-22% on the v8.3 shipping binaries relative to v8.2 with no remaining fix in the server. The cause is the default compiler switch to Clang (SERVER-107041), not application code. BF-45937 reduced the overall v8.3-vs-v8.2 percentile regression to this compiler effect; this ticket tracks the toolchain-side remediation for the portion that cannot be fixed in the server.
Background
BF-45937 ("[v8.3 vs v8.2] Regression in OperationThroughput on percentiles_window and percentiles_expr") root-caused the regression to SERVER-107041 flipping the default package/archive_dist_test compiler to Clang (+ ThinLTO + PGO) for v8.3 and master. v8.2 built with plain GCC opt.
Evidence from the BF-45937 investigation, all on identical v8.3 source head (c9a04e2371d):
- Removing LTO/PGO while staying on Clang moves percentile_tdigest only +1.2%.
- GCC vs 8.2 reaches parity on every BF workload, e.g. percentile_tdigest -0.9% (classic engine) and array_10_percentile_many100 -0.3% (classic engine).
- Clang is ~15-22% slower than GCC on the percentile paths: CPU usage of GCC vs Clang is +18.4% on percentile_tdigest and +21.9% on array_10_percentile_many100 (classic).
- The cost concentrates in two codegen patterns:
- the boost::optional/tassert round-trip in representAsChecked() inside computeDiscreteRank() (src/mongo/db/pipeline/percentile_algo.h), which drives the array / discrete-rank path (percentiles_expr);
- the TDigest accumulate loop driving the percentile_tdigest window phases (percentiles_window).
- Clang is also faster than GCC elsewhere: scalar percentiles +19-31% vs 8.2, bulk insert (CreateDataset) +11%. A blanket compiler revert is therefore not obviously correct.
See this perf patch: https://spruce.corp.mongodb.com/version/6a99acda148f9c0007fd8791/tasks?sorts=STATUS%3AASC%3BBASE_STATUS%3ADESC for proof that switching the compiler to GCC improves the perf.
Options
- File an LLVM/Clang codegen report for the representAsChecked/boost::optional and TDigest-accumulate patterns and track a compiler-side fix.
- Tune Clang flags and/or the PGO training-profile coverage for these workloads.
- Accept the perf regression since its a compiler issue.
- is related to
-
SERVER-107041 Switch Server over to Clang for Opt Builds if Appropriate
-
- Closed
-
-
SERVER-126799 Filter NaN from $percentile/$median window function inputs
-
- Closed
-