Add compound index to cover findUnaggregatedSettledDocsUpTo query

XMLWordPrintableJSON

    • Atlas SQL
    • Not Needed

      Follow-up from PR #179651 (SQL-3328).

      Problem

      The aggregateDailyUsage activity's query, SqlInterfaceStatusDao.findUnaggregatedSettledDocsUpTo, filters the SqlInterfaceStatus collection on:

      { dailyStatus: { $exists: false },
        date: { $lte: <yesterday> },
        events.propagation_status: { $ne: "PENDING" } }
      

      None of the three existing indexes can serve this query:

      • (clusterId, date) (unique) — leading field is clusterId, so it cannot range-scan on date alone
      • (meterUsageId) (unique) — unrelated
      • (submittedAt) (TTL) — unrelated

      As a result MongoDB performs a full collection scan (COLLSCAN) on every daily activity run. At scale (many clusters, one document per cluster per day) this scan grows unbounded within the 180-day TTL window.

      Proposed fix

      Add a compound index on (dailyStatus, date) via a new BaseCreateIndexMigration (mirroring CreateSqlInterfaceStatusIndexesMigration), then bring the gov backing databases to parity as that migration already does.

      Rationale:

      • dailyStatus as the leading field lets the planner seek the "absent" bucket (the query targets docs without a dailyStatus).
      • date as the second field supports the $lte range bound.
      • events.propagation_status ($ne on a multikey array) is not selective and cannot contribute a tight index bound, so it is left as a residual post-filter.

      Acceptance criteria

      • New index (dailyStatus, date) created via migration, with explicit name matching live commercial naming conventions (idempotent by name).
      • explain() on findUnaggregatedSettledDocsUpTo shows an IXSCAN (not COLLSCAN).
      • Idempotency integration test added, matching the existing CreateSqlInterfaceStatusIndexesMigrationIdempotencyIntTests pattern.

      References

      • PR #179651 review comment (nbagnard + Augment)
      • systems/sql-interface/migration/src/main/java/com/xgen/sqlinterface/migration/CreateSqlInterfaceStatusIndexesMigration.java

            Assignee:
            Neal Beeken
            Reporter:
            Jonathan Chemburkar
            Jonathan Chemburkar
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: