• Type: New Feature
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • DB Integration & Observability
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Overview

      Build the golden test framework that runs an identical workload against a last-LTS binary and a current-tree binary and asserts the parent's serverStatus output is shape-compatible with the current output (epic SPM-4962).

      Scope of Work

      * Provisioning: buildscripts/metrics_golden/provision_last_lts.sh (resolve last LTS via multiversion config, symlink binaries under metrics-golden-last-lts), a new metrics_golden_shape Evergreen task, and a metrics_golden resmoke suite selecting jstests/observability_golden/*/.js.
      * Shape extraction: jstests/libs/observability/metrics_shape.js with shapeOf (type-only skeleton, arrays collapsed to the union of element shapes) preserving exact BSON leaf types (NumberInt/NumberLong/NumberDouble), and compatDiff (every path in the old shape must exist with the same type; new paths reported separately).
      * Runner: metrics_golden_harness.js with runGoldenWorkload: start each topology twice (parent then current binaries), run harness and workload warm-up, assert same-binary self-stability of captures, take the baseline after warm-up, compare shapes, then compare per-binary delta values. Default validator is exact delta equality with prefix-matched exempt and rel/abs tolerant lists.
      * Workload format: plain JS tests with required covers, serverParams, topology, sections, and setup/warmup/ops phases.
      * Enforcement: buildscripts/idl/check_metrics_golden_workload_coverage.py fails the build when a version-declared IDL family has no covering golden workload, and check_lts_versions.py writes the skip list for families whose declared version differs between the two trees.

      Design source

      [Technical Design: Unified Metrics Framework|$GDOC], Validation Framework tab (entire).

            Assignee:
            Unassigned
            Reporter:
            Finley Lau
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: