Context

Follow-up to WT-18640 for the Storage Engines health dashboards:

• Single Cluster
• Fast Truncate Fleet Health

At least three issues remain: intermittent query failures, long explicit cluster-ID filters, and duplicated health-check definitions. These are expected to become more severe as we extend the dashboards to cover more components (at time of writing only Fast Truncate is covered, with 6 health checks).

Current issues

• Intermittent 503s and timeouts. Queries sometimes succeed individually but fail during concurrent dashboard loading. Observed errors include connection refused, connection reset, “no healthy upstream”, and timeouts awaiting response headers. Even a vector(1) probe has failed during some incidents, so query complexity alone does not explain the failures; the backend root cause is unconfirmed.

• Long explicit cluster lists. Some fleet queries expand the hidden cluster_id variable into a regex containing every discovered cluster ID. This produces large requests (100+ concatenated cluster ids) and couples query scope to variable discovery. Removing the filter has previously caused broad historical scans and memory-limit errors.
• Duplicated query definitions. Health calculations are copied across current/history summaries, attention tables, version/tier breakdowns, and single-cluster diagnostic panels. Changes require updating multiple copies, creating a risk of inconsistent thresholds, sampling windows, and availability handling. Currently there isn't known way to reuse results or queries from panels across different dashboards, but reusing queries within the same dashboard is possible: Prototype

Scope

1. Investigate the 503s and timeouts. Determine whether failures originate from service availability, routing, query resource pressure, or a combination.
2. Replace long explicit cluster-ID filters with a bounded filtering approach that preserves the intended fleet population (Infinite clusters) without introducing expensive broad scans.
3. Establish a maintainable source for shared health-check definitions and validate consistency across dashboards.