-
Type:
Task
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: Atlas SQL Interface
-
Atlas SQL
-
Not Needed
Follow-up from PR #179651 (SQL-3328).
Problem
The aggregateDailyUsage activity's query, SqlInterfaceStatusDao.findUnaggregatedSettledDocsUpTo, filters the SqlInterfaceStatus collection on:
{ dailyStatus: { $exists: false },
date: { $lte: <yesterday> },
events.propagation_status: { $ne: "PENDING" } }
None of the three existing indexes can serve this query:
- (clusterId, date) (unique) — leading field is clusterId, so it cannot range-scan on date alone
- (meterUsageId) (unique) — unrelated
- (submittedAt) (TTL) — unrelated
As a result MongoDB performs a full collection scan (COLLSCAN) on every daily activity run. At scale (many clusters, one document per cluster per day) this scan grows unbounded within the 180-day TTL window.
Proposed fix
Add a compound index on (dailyStatus, date) via a new BaseCreateIndexMigration (mirroring CreateSqlInterfaceStatusIndexesMigration), then bring the gov backing databases to parity as that migration already does.
Rationale:
- dailyStatus as the leading field lets the planner seek the "absent" bucket (the query targets docs without a dailyStatus).
- date as the second field supports the $lte range bound.
- events.propagation_status ($ne on a multikey array) is not selective and cannot contribute a tight index bound, so it is left as a residual post-filter.
Acceptance criteria
- New index (dailyStatus, date) created via migration, with explicit name matching live commercial naming conventions (idempotent by name).
- explain() on findUnaggregatedSettledDocsUpTo shows an IXSCAN (not COLLSCAN).
- Idempotency integration test added, matching the existing CreateSqlInterfaceStatusIndexesMigrationIdempotencyIntTests pattern.
References
- PR #179651 review comment (nbagnard + Augment)
- systems/sql-interface/migration/src/main/java/com/xgen/sqlinterface/migration/CreateSqlInterfaceStatusIndexesMigration.java