Improve testing of Histograms

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Networking & Observability
    • N&O Prioritized List
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Summary
      Clarify the canonical histogram format for serverStatus metrics and improve cross-team test coverage to prevent silent breakages.

      Context

      SERVER-121260 changed how disagg.pageServer.GetPageAtLSNStreaming.latencyMicros emits histogram data. The new format encodes bucket boundaries in metric names (e.g. metric__250_500.count) rather than using the existing {{

      {lowerBound: Long, count: Long}

      }} array structure that mms-automation already knows how to parse into OTel histograms.

      This broke the InTel collection pipeline. The agent has no signal that these are histograms, so it treats each bucket as a flat counter. The metrics arrive in Grafana as individual time series, unusable by histogram_quantile.

      The change was made in good faith. The author checked for references to the old counters but was unaware of the regex-based histogram detection in mongometrics_collector.go and histogram_utils.go.

      There are now at least three competing histogram representations in serverStatus: the {{

      {lowerBound, count}}} array format, the {micros, count} format, and this new vmrange-style flat counter format. Each has different tradeoffs around FTDC compression, T2 readability, and downstream parsability.

      Risks

      Future histogram additions will likely repeat this breakage unless the expected format is documented and tested outside the server repo.

      The vmrange format is VictoriaMetrics-specific. Teams shipping to multiple backends (InTel, MaaS, others) cannot easily consume it without custom handling per destination.

      Scope
      # Document the canonical histogram format for serverStatus. Capture the tradeoffs between cumulative (le), {{{lowerBound, count}

      }} arrays, and vmrange-style flat counters. Agree on what new instrumentation should use going forward, or explicitly define when each is appropriate.

      1. Add tests server repo that validate histogram parsing end-to-end for at least the known formats.
      2. Investigate whether mongometrics_collector.go needs to be extended to detect the new flat counter format, or whether SERVER-121260 should be reverted to match the existing {{ {lowerBound, count}

        }} structure.

            Assignee:
            Unassigned
            Reporter:
            Jonas Bergler
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: