OTel file metric exporter writes an unbounded metrics.json, breaking shell cat() (16 MB limit) and failing otel_metrics.js

XMLWordPrintableJSON

    • Query Optimization
    • Fully Compatible
    • ALL
    • 200
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      The disagg-storage otel_metrics.js test fails on the TSAN all-feature-flags variant with:

          Error: cat() : file .../mongodb-<pid>-<date>-metrics.jsonl too big to load as a variable (file is 16898149 bytes, limit is 16777216 bytes.)
          readJsonlFile@jstests/noPassthrough/observability/libs/otel_file_export_helpers.js:52
          getLatestRawRecord ... getLatestMetrics ... getGaugeOrCounterMetricValue (assert.soon)
          @ src/mongo/db/modules/atlas/jstests/disagg_storage/otel_metrics.js
          exiting with code -3
      

      Root Cause. The OpenTelemetry OTLP file metric exporter appends one record per export interval to a single per-day file. It is configured in src/mongo/otel/metrics/metrics_initialization.cpp
      (initializeFile()) with the pattern:

      {directory}/mongodb-{pid}-%Y%m%d-metrics.jsonl
      

      This pattern has no rotation-index placeholder (%N), which defeats the exporter's built-in size-based rotation (library defaults: file_size=20 MB, rotate_size=3). As a result the file grows without bound within a day.

      The mongo shell's cat() builtin refuses to load any file >= 16 MB (kFileSizeLimit = 16*1024*1024 in src/mongo/shell/shell_utils_extended.cpp). Test helpers read these files via cat(). Under slow TSAN execution the server stays up long enough for the file to cross 16 MB, so cat() throws and the test aborts. Note the default 20 MB cap is itself above the 16 MB shell limit, so even with rotation working the file would still be
      unreadable.

            Assignee:
            Matt Olma
            Reporter:
            Matt Olma
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: