percentile aggregations regress due to default compiler switch to clang

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • DevProd Build
    • 200
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Problem

      $percentile / $median aggregation hot paths regress ~16-22% on the v8.3 shipping binaries relative to v8.2 with no remaining fix in the server. The cause is the default compiler switch to Clang (SERVER-107041), not application code. BF-45937 reduced the overall v8.3-vs-v8.2 percentile regression to this compiler effect; this ticket tracks the toolchain-side remediation for the portion that cannot be fixed in the server.

      Background

      BF-45937 ("[v8.3 vs v8.2] Regression in OperationThroughput on percentiles_window and percentiles_expr") root-caused the regression to SERVER-107041 flipping the default package/archive_dist_test compiler to Clang (+ ThinLTO + PGO) for v8.3 and master. v8.2 built with plain GCC opt.

      Evidence from the BF-45937 investigation, all on identical v8.3 source head (c9a04e2371d):

      • Removing LTO/PGO while staying on Clang moves percentile_tdigest only +1.2%.
      • GCC vs 8.2 reaches parity on every BF workload, e.g. percentile_tdigest -0.9% (classic engine) and array_10_percentile_many100 -0.3% (classic engine).
      • Clang is ~15-22% slower than GCC on the percentile paths: CPU usage of GCC vs Clang is +18.4% on percentile_tdigest and +21.9% on array_10_percentile_many100 (classic).
      • The cost concentrates in two codegen patterns:
        • the boost::optional/tassert round-trip in representAsChecked() inside computeDiscreteRank() (src/mongo/db/pipeline/percentile_algo.h), which drives the array / discrete-rank path (percentiles_expr);
        • the TDigest accumulate loop driving the percentile_tdigest window phases (percentiles_window).
      • Clang is also faster than GCC elsewhere: scalar percentiles +19-31% vs 8.2, bulk insert (CreateDataset) +11%. A blanket compiler revert is therefore not obviously correct.

      See this perf patch: https://spruce.corp.mongodb.com/version/6a99acda148f9c0007fd8791/tasks?sorts=STATUS%3AASC%3BBASE_STATUS%3ADESC for proof that switching the compiler to GCC improves the perf.

      Options

      1. File an LLVM/Clang codegen report for the representAsChecked/boost::optional and TDigest-accumulate patterns and track a compiler-side fix.
      2. Tune Clang flags and/or the PGO training-profile coverage for these workloads.
      3. Accept the perf regression since its a compiler issue.

            Assignee:
            Unassigned
            Reporter:
            Finley Lau
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: