Create Grafana dashboard to view index build fleet-wide metrics

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Storage Execution
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      This dashboard should give us insight into the health of index builds that customers are running and the ability to proactively address problematic index builds. Part of the work here would also be to define which metrics we care about tracking.

      Some insights that could be valuable include:

      • Index Build Failures (by type)
      • Abnormal Index Build duration (this would involve first creating a way to estimate index build duration by collection size and defining threshold for what is considered longer than expected)
      • The duration of index build phases that block replication (metrics added in SERVER-114363)
      • Index build spilling

      Being able to track some of this may involve creating/whitelisting new metrics. The current whitelisted metrics are (found here):

      • "mongodb.serverStatus.metrics.commands.voteAbortIndexBuild.total"
      • "mongodb.serverStatus.metrics.commands.voteCommitIndexBuild.total
      • "mongodb.serverStatus.indexBuilds.total"
      • "mongodb.serverStatus.indexBuilds.phases.commit"
      • "mongodb.serverStatus.indexBulkBuilder.numSorted"
      • "mongodb.serverStatus.indexBulkBuilder.bytesSorted"
      • "mongodb.serverStatus.indexBulkBuilder.bytesSpilled"
      • "mongodb.serverStatus.indexBulkBuilder.bytesSpilledUncompressed"
      • "mongodb.serverStatus.indexBulkBuilder.memUsage"
      • "mongodb.serverStatus.indexBulkBuilder.spilledRanges"

      Example Grafana Dashboard

            Assignee:
            Unassigned
            Reporter:
            Damian Wasilewicz
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: