[SERVER-57006] Delete unneeded documents that have been spilled to disk in window functions Created: 17/May/21  Updated: 06/Dec/22

Status: Backlog
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Louis Williams Assignee: Backlog - Query Optimization
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Assigned Teams:
Query Optimization
Participants:

 Description   

When window functions spill to disk, they sometimes are able to free unneeded records.

I originally advised that we not delete records because this can be impactful on the WT cache, however, this assumption was based on normal MongoDB writes. The SpillableCache has special cases where deleting old records may be beneficial:

  • These temporary tables are not journaled, so there is no write amplification by deleting old records.
  • We don't timestamp writes to these tables, so all writes are globally visible. As a result, when WT does take a checkpoint, it only needs to store one version of each document. Deleting unneeded records reduces the amount of data written to disk during checkpoints.

CC ted.tuckman nicholas.zolnierz



 Comments   
Comment by Eric Milkie [ 17/May/21 ]

Bear in mind that deleting documents after they have been checkpointed will introduce needless extra I/O if another checkpoint runs before the table is dropped. It's not a clear win to do this optimization; it depends how long-lived these tables are. Note that the default checkpoint frequency in MongoDB is 60 seconds.

Generated at Thu Feb 08 05:40:44 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.