-
Type:
Task
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Networking & Observability
-
N&O Prioritized List
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Summary
Clarify the canonical histogram format for serverStatus metrics and improve cross-team test coverage to prevent silent breakages.
Context
SERVER-121260 changed how disagg.pageServer.GetPageAtLSNStreaming.latencyMicros emits histogram data. The new format encodes bucket boundaries in metric names (e.g. metric__250_500.count) rather than using the existing {{
}} array structure that mms-automation already knows how to parse into OTel histograms.
This broke the InTel collection pipeline. The agent has no signal that these are histograms, so it treats each bucket as a flat counter. The metrics arrive in Grafana as individual time series, unusable by histogram_quantile.
The change was made in good faith. The author checked for references to the old counters but was unaware of the regex-based histogram detection in mongometrics_collector.go and histogram_utils.go.
There are now at least three competing histogram representations in serverStatus: the {{
{lowerBound, count}}} array format, the {micros, count} format, and this new vmrange-style flat counter format. Each has different tradeoffs around FTDC compression, T2 readability, and downstream parsability.Risks
Future histogram additions will likely repeat this breakage unless the expected format is documented and tested outside the server repo.
The vmrange format is VictoriaMetrics-specific. Teams shipping to multiple backends (InTel, MaaS, others) cannot easily consume it without custom handling per destination.
Scope
# Document the canonical histogram format for serverStatus. Capture the tradeoffs between cumulative (le), {{{lowerBound, count}
}} arrays, and vmrange-style flat counters. Agree on what new instrumentation should use going forward, or explicitly define when each is appropriate.
- Add tests server repo that validate histogram parsing end-to-end for at least the known formats.
- Investigate whether mongometrics_collector.go needs to be extended to detect the new flat counter format, or whether
SERVER-121260should be reverted to match the existing {{ {lowerBound, count}}} structure.
- is related to
-
SERVER-121260 Change output format of HistogramServerStatusMetric to be t2 friendly
-
- Closed
-
-
SERVER-129802 Revert SERVER-121260
-
- Closed
-