-
Type:
Bug
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: None
-
None
-
Storage Execution
-
Fully Compatible
-
ALL
-
Storage Execution 2026-11-09
-
0
-
None
-
None
-
None
-
None
-
None
-
None
-
None
PrimaryDrivenResumableIndexBuildTest._readResumeMetrics() in jstests/noPassthrough/libs/index_builds/primary_driven.js assumes one OTel metrics file per node and sums the last cumulative snapshot of every file in the metrics directory. However, the OTel file exporter rotates files for a single node:
- at midnight UTC, because the file name embeds the date
(mongodb-<pid>%Y%m%d%N-metrics.jsonl, see
src/mongo/otel/metrics/metrics_initialization.cpp) - when a file reaches the 8MB chunk limit (the %N rotation index)
Every rotated file retains its last cumulative snapshot, so after a rotation a single node's counter is summed once per file and the cluster-wide total is inflated. This makes the resume metric assertions in the primary-driven index build test library flaky for any run that crosses midnight UTC or triggers the chunk limit.
Suggested fix
Group the files by pid (parsed from the file name) and take the per-phase maximum across a pid's files as that node's current value. The counters are cumulative within a process, so the max equals the newest snapshot, and it also tolerates reading the directory right after a rotation when the new file may not have been flushed yet. Then sum across pids.
Related work
SERVER-127030 tracks replacing this reader with getLatestMetrics() or an equivalent once attributes are preserved there. That work is about attribute preservation and does not by itself address the rotation double-count described here. Any replacement reader must still aggregate the latest snapshot per node rather than summing across files or taking a single global snapshot.
- is related to
-
SERVER-138511 OTel metrics file export helpers double-count a node after exporter file rotation
-
- Backlog
-
- related to
-
SERVER-127030 Use getLatestMetrics or equivalent in PDIB testing library
-
- Blocked
-